Introduction

There are patterns that you use without realizing you are actually using a pattern—patterns with defined pros, cons, and recommendations for when to use them. Choreography is one such pattern for me. For many years, I used it simply as part of my day-to-day development activities, never realizing it was a pattern. So, let's dig deeper into choreography.

What Is the Choreography Pattern?

A common analogy for this pattern is dancers: all of them move in the correct rhythm, following each other's movements. This seems to be the main idea behind the name. Let's translate this simple analogy into the world of microservices. If we replace dancers with services, we get the following: all services must do their work independently, following each other without a central coordinator.

Let's look at this with a small example. Suppose we have a web shop where customers buy goods. When someone places an order, we should:

• Create an order

• Send a notification to the buyer that the order was successfully created

• Book inventory in the warehouse

• Create a delivery

• And so on

This part of the flow is enough to represent the idea behind the choreography pattern.

At first, let's imagine how we can design this system using a monolithic approach.

The diagram above is straightforward. The user makes a request to create an order, and a set of services are called one after another:

• OrderService creates the order.

• InventoryService books the inventory in the warehouse.

• EmailService sends the buyer an email confirming the order was created.

• DeliveryService creates the delivery so the delivery flow can start.

The crucial part of this diagram is the order in which we call each service. We cannot book inventory in the warehouse or send an email notification before the order is created, so we wait until the order is created. The delivery cannot be created if we do not have the right quantity of products in the warehouse, so we need to know that the inventory was booked. OrderController acts as an orchestrator for this flow, knowing when and which services should be called. If an exception occurs for some reason, the next service won't be called. This implementation is synchronous and tightly coupled—we know about every step and follow them.

Let's translate this design to a microservices architecture. It is clear that almost every service from the diagram can be represented as a separate microservice, and OrderController will be merged into OrderService. The design of the system might look like this:

What catches the eye at first glance is the dependencies. The OrderService not only knows about other services but also completely depends on them. So, we have a tightly coupled, fragile design. What we need to do is remove direct communication between our microservices, and events can help with this.

I have already hinted at some possible events when we considered the monolithic implementation of the workflow. Let's define the full list:

• OrderCreated is published when the order is created.

• InventoryBooked is published when the inventory is booked in the warehouse.

• DeliveryCreated is published when the delivery is created for the order.

Now we can redesign our system using event-driven microservices.

There is no direct connection between services in the new diagram. The message broker (I use Azure Service Bus) receives and delivers all messages. Let's cover how the entire flow works now:

• The user makes a request to create an order through OrderService.

• OrderService publishes the OrderCreated event. At this step, the service has completed its part of the flow and simply waits for new requests.

• The Service Bus delivers the OrderCreated event to InventoryService and EmailService. Both services can start processing this event simultaneously because their work results do not depend on each other.

• When InventoryService finishes booking the inventory for the order, it publishes the InventoryBooked event.

• The Service Bus delivers the InventoryBooked event to DeliveryService.

• When DeliveryService finishes, it publishes the DeliveryCreated event.

• And so on.

We can continue adding many different services to this flow, but I assume the main idea is clear. Our services are independent participants that perform their own tasks in the overall operation. This is an example of the Choreography pattern.

Problems and Considerations

Simplicity of Extension

The Choreography pattern looks very simple and useful: one service publishes events, and other services subscribe to and process these events. For example, in the design above, we can simply add a few services and a few additional events:

• EmailService can subscribe to DeliveryCreated event and send an email with delivery information.

• An additional InvoiceService can subscribe to the OrderCreated event to generate an invoice.

• And so on.

Choreography shines when we have a small number of services and everything works as expected. However, as the workflow grows, we can experience painful issues with finding and fixing defects.

Event Ordering

Choreography does not define an explicit order of events. We can manage this only by introducing new events and changing the flow to work with them. This requires many changes and coordination between teams if new events must be processed by services outside your area of responsibility. Just imagine that you need to book inventory only when the order has already been paid.

We already have the OrderCreated event, so we add a new subscription for the PaymentService to process the payment. Otherwise, InventoryService will not account for this new rule. To do this, we need to introduce a new OrderPaid event and change the implementation of inventory booking.

What if many services need to wait until the order is paid? This pattern does not guarantee the order of events. It is still possible to create pseudo-ordering, but if you need to do this, perhaps you need another pattern, such as Orchestration or Saga.

Eventual Consistency

Choreography leads to eventual consistency by design. We do not know how quickly the same order will be processed by different systems. For example, a user might receive an email that a delivery was created before the delivery tracking system has processed that information. So, when the user opens the link, they receive an HTTP 404 instead of delivery information.

Debugging Difficulty

Let's imagine that a delivery was not created for an order. We need to check if there are issues with the DeliveryService. If it looks fine, we need to check if the order was initially created and whether the inventory was booked. This does not seem so bad, right?

What about a slightly changed flow of events:

OrderCreated → OrderPaid → InvoiceGenerated → InventoryBooked → InventoryPacked → DeliveryCreated

A support ticket stating that a delivery was not created would take much more time to investigate. We need to check that all five additional events were processed properly.

This is why it is crucial for such systems to have good log analytics software. The necessary logs should be collected and stored. Each event should have a special identifier for the flow, usually called a CorrelationId, which is generated with the first event. Without this identifier, it is nearly impossible to follow all events in a distributed system.

Processing Dead Letters

Another important consideration is the notification system. When an event cannot be processed properly by a service, it is moved to a Dead-Letter Queue (DLQ) by the message broker. With a proper notification mechanism, this might be the first indicator that the system is experiencing issues. This allows the development team to quickly recover the system and reprocess all events from the DLQ.

Use Outbox for Reliable Event Publishing

The core of Choreography is event handling. Microservices publish and listen to events. Therefore, we need to ensure that events are published reliably. For this, the Outbox pattern can be used. I have already published an article, "Outbox Pattern in Microservices: Reliable Event Publishing in C#," about it.

Event Schema and Versioning

As the system evolves, event schemas change. It is important to ensure that you support different versions.

It is also important to have a schema registry where members of each team can look up events and their supported versions. You cannot simply deprecate an event version without notifying all teams in advance, because as long as at least one consumer uses it, you must support that version.

Idempotency

The same event can be sent multiple times due to network issues or retries, and consumers should be ready for this. There are different options for handling duplicate events, but the main idea is simple: the consumer should either apply the same changes multiple times or skip processing the duplicated event.

Performance

Choreography can lead to a high volume of messages. In high-load systems, the message broker might become a bottleneck and increase the cost of the system.

Testing

Testing such a system, where many events arrive at many services and might be processed almost simultaneously, is really challenging. You need to:

• Create many different events.

• Simulate event failures to check how the system behaves.

• Verify that if an event is processed out of order, the system does not enter an incorrect state.

• Check the handling of events in the DLQ.

• And so on.

When to Use

Given the problems we have already outlined, the Choreography pattern is a useful tool in the right hands. Let's look at the cases where it is best to use it.

Small Number of Events

As we have already seen, a system with a few events can easily use this pattern. With proper log analytics and retry mechanisms, even big issues should not be a huge problem.

Consumers from Other Sub-Domains

It is a common situation when an event needs to be consumed outside your sub-domain. In such cases, the system that needs to process the event is likely developed and maintained by another team.

Look at our list of services, and you can see the possible sub-domains or teams responsible for each service:

• Ordering: OrderService

• Warehouse: InventoryService

• Platform: EmailService

• Delivery: DeliveryService

When Not to Use

I initially wanted to say, "Do not use in all other situations," but I decided it is better to highlight some specific cases where Choreography seems like the right choice but is not.

Complex Workflows

With many steps in your workflow, choreography is really hard to manage. Just imagine a situation where, after a payment failure, you need to issue a refund, or a product is no longer available in the warehouse and you need to cancel all orders, issue refunds, and send emails to buyers with explanations. This might be a good case for Orchestration, not Choreography.

Strong Consistency Requirements

I have already noted that Choreography is all about eventual consistency. We send events, and we do not know when they will be processed by consumers. If the system you are working on requires strong consistency, do not use Choreography.

Imagine a situation where User1 sends money to User2. At some point, we might have a situation where User2 has already received the money, but it has not yet been withdrawn from User1. So, User1 can still spend it even after transferring it to another user.

Implementation

I will use Azure Functions and Azure Service Bus for the examples. As a base for the implementation, I will use the same flow we have already discussed. After some changes, it should look like this:

The code snippets below do not represent production-quality code. Some fake services are used just to outline the main idea behind the Choreography pattern.

Let's start with the CreateOrder function. Its source code is in the snippet below:

public class CreateOrder(
    ServiceBusClient serviceBusClient,
    FakeOrderStore orderStore,
    ILogger<CreateOrder> logger)
{
    private static readonly string[] Products = ["Laptop", "Mouse", "Keyboard", "Monitor", "Headphones"];
    private static readonly string[] Emails = ["alice@example.com", "bob@example.com", "carol@example.com"];

    [Function(nameof(CreateOrder))]
    public async Task<HttpResponseData> Run(
        [HttpTrigger(AuthorizationLevel.Anonymous, "get")] HttpRequestData req)
    {
        var correlationId = Guid.NewGuid();
        var orderId = Guid.NewGuid();
        var product = Products[Random.Shared.Next(Products.Length)];
        var email = Emails[Random.Shared.Next(Emails.Length)];
        var amount = Random.Shared.Next(1, 4);

        var orderEvent = new OrderCreated
        {
            CorrelationId = correlationId,
            OrderId = orderId,
            CustomerEmail = email,
            Product = product,
            Amount = amount
        };

        orderStore.Add(orderEvent);

        var sender = serviceBusClient.CreateSender("order-events");
        var message = new ServiceBusMessage(JsonSerializer.SerializeToUtf8Bytes(orderEvent))
        {
            Subject = "OrderCreated"
        };
        message.ApplicationProperties.Add("CorrelationId", correlationId.ToString());
        await sender.SendMessageAsync(message);

        logger.LogInformation(
                "[{CorrelationId}] OrderCreated event published: OrderId={OrderId}",
                orderEvent.CorrelationId,
                orderEvent.OrderId);

        var response = req.CreateResponse(System.Net.HttpStatusCode.OK);
        await response.WriteAsJsonAsync(orderEvent);
        return response;
    }
}

The code is straightforward and includes a mock implementation for creating a fake order. The most interesting part is in lines, where we create a sender serviceBusClient.CreateSender for the topic named order-events, create a ServiceBusMessage, and set the CorrelationId. After sending the event, the entire flow starts.

The BookInventory function works at the warehouse level and books the necessary inventory for the order. If there are not enough products in stock, the function does not send the OrderInventoryBooked event.

public class BookInventory(
    ServiceBusClient serviceBusClient,
    FakeInventoryStore inventoryStore,
    ILogger<BookInventory> logger)
{
    [Function(nameof(BookInventory))]
    public async Task Run(
        [ServiceBusTrigger("order-events", "order-created-inventory-sub", Connection = "ServiceBusConnection")]
        ServiceBusReceivedMessage message)
    {
        var orderEvent = JsonSerializer.Deserialize<OrderCreated>(message.Body.ToStream())!;

        if (inventoryStore.TryBook(orderEvent.Product, orderEvent.Amount))
        {
            logger.LogInformation(
                "[{CorrelationId}] BookInventory: Inventory booked for order {OrderId} (product: {Product}, qty: {Amount})",
                orderEvent.CorrelationId,
                orderEvent.OrderId,
                orderEvent.Product,
                orderEvent.Amount);

            var bookedEvent = new OrderInventoryBooked
            {
                CorrelationId = orderEvent.CorrelationId,
                OrderId = orderEvent.OrderId,
                CustomerEmail = orderEvent.CustomerEmail,
                Product = orderEvent.Product,
                Amount = orderEvent.Amount
            };

            var sender = serviceBusClient.CreateSender("order-events");
            var bookedMessage = new ServiceBusMessage(JsonSerializer.SerializeToUtf8Bytes(bookedEvent))
            {
                Subject = "OrderInventoryBooked"
            };
            bookedMessage.ApplicationProperties.Add("CorrelationId", orderEvent.CorrelationId.ToString());
            await sender.SendMessageAsync(bookedMessage);

            logger.LogInformation(
                "[{CorrelationId}] OrderInventoryBooked published: OrderId={OrderId}",
                orderEvent.CorrelationId,
                orderEvent.OrderId);
        }
        else
        {
            logger.LogWarning(
                "[{CorrelationId}] BookInventory: Failed to book inventory for order {OrderId} (product: {Product}, qty: {Amount}) - insufficient stock",
                orderEvent.CorrelationId,
                orderEvent.OrderId,
                orderEvent.Product,
                orderEvent.Amount);
        }
    }
}

The function has a ServiceBusTrigger with a subscription for the order-events topic. The topic receives many different order events, not only those related to creation. Therefore, the subscription uses the expression sys.Label = 'OrderCreated' to filter out only the necessary ones.

You can see that we first try to book the inventory inventoryStore.TryBook, and if we succeed, a new OrderInventoryBooked event is published to the same order-events topic.

In parallel with BookInventory, the SendEmail function notifies the buyer with an order confirmation.

public class SendEmail(ILogger<SendEmail> logger)
{
    [Function(nameof(SendEmail))]
    public void Run(
        [ServiceBusTrigger("order-events", "order-created-email-sub", Connection = "ServiceBusConnection")]
        ServiceBusReceivedMessage message)
    {
        var orderEvent = JsonSerializer.Deserialize<OrderCreated>(message.Body.ToStream())!;

        logger.LogInformation(
            "[{CorrelationId}] SendEmail: Email sent to {Email} for order {OrderId}",
            orderEvent.CorrelationId,
            orderEvent.CustomerEmail,
            orderEvent.OrderId);
    }
}

The last function in our workflow is CreateDelivery.

public class CreateDelivery(
    ServiceBusClient serviceBusClient,
    FakeDeliveryStore deliveryStore,
    ILogger<CreateDelivery> logger)
{
    [Function(nameof(CreateDelivery))]
    public async Task Run(
        [ServiceBusTrigger("order-events", "inventory-booked-delivery-sub", Connection = "ServiceBusConnection")]
        ServiceBusReceivedMessage message)
    {
        var bookedEvent = JsonSerializer.Deserialize<OrderInventoryBooked>(message.Body.ToStream())!;

        var deliveryId = Guid.NewGuid();
        var deliveryEvent = new DeliveryCreated
        {
            CorrelationId = bookedEvent.CorrelationId,
            OrderId = bookedEvent.OrderId,
            DeliveryId = deliveryId
        };

        deliveryStore.Add(deliveryEvent);

        var sender = serviceBusClient.CreateSender("delivery-events");
        var deliveryMessage = new ServiceBusMessage(JsonSerializer.SerializeToUtf8Bytes(deliveryEvent))
        {
            Subject = "DeliveryCreated"
        };
        deliveryMessage.ApplicationProperties.Add("CorrelationId", bookedEvent.CorrelationId.ToString());
        await sender.SendMessageAsync(deliveryMessage);

        logger.LogInformation(
            "[{CorrelationId}] CreateDelivery: Delivery created for order {OrderId}, DeliveryId={DeliveryId}",
            bookedEvent.CorrelationId,
            bookedEvent.OrderId,
            deliveryId);

        logger.LogInformation(
            "[{CorrelationId}] DeliveryCreated published to delivery-events: OrderId={OrderId}",
            bookedEvent.CorrelationId,
            bookedEvent.OrderId);
    }
}

Here we see the same behavior as in the previous functions: first, we process the event, and in response, we publish another event indicating a successful result.

The full source code can be found on GitHub.

Summary

The Choreography pattern is usually the first step into event-driven architecture. What could be simpler than publishing an event, executing code upon receipt, and publishing another event as a response? Simplicity and flexibility are key, but with these come many considerations and problems that need to be addressed, such as DLQ maintenance, notifications, eventual consistency, debugging, and testing.

Links

Choreography Pattern - Azure Architecture Center

Azure Service Bus trigger for Azure Functions

Cover Image by Evgen Rom from Pixabay