# Orchestration Pattern in Microservices

# Introduction

In my previous article about [Choreography](https://iharmaiseyeu.net/choreography-pattern-in-microservices), I outlined that it does not suit situations with complicated workflows or where strong consistency is required. So, it was logical to discuss Orchestration, the pattern that helps solve - if not all - many of the issues of Choreography.

# What Is the Orchestration Pattern?

If we compare Choreography to dancers following each other, we can imagine Orchestration as an orchestra following a conductor who knows where and when each musician should start their part. The main idea behind this pattern is to have one service that understands the business process and the order in which operations should be called.

When we talk about the business process, we do not mean that the orchestrator should implement the business logic. On the contrary, it should not implement any business logic—only call the services that implement parts of the business logic, check the results, and decide whether the process should continue.

The diagram below outlines a simple orchestration process.

![](https://cdn.hashnode.com/uploads/covers/64c01f633d0d5beec2f4cb9f/928764ad-662b-4acd-bfa5-655ecec240c9.svg align="center")

![](https://docs.2getherme.nl/api/files.get?sig=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJrZXkiOiJ1cGxvYWRzLzQ0NmZkYTIzLTM5MmItNGZmOS1hYjE3LWQ3NjJjOGI3YjI0Zi85ZGM2YTEzNC05N2FjLTQyYzMtOWI1OS03MzIyMDkwZjhhMGUvZGlhZ3JhbS5zdmciLCJ0eXBlIjoiYXR0YWNobWVudCIsImlhdCI6MTc4NDgzMDcxMSwiZXhwIjoxNzg1NDM1NTExfQ.eJO3I08U5WkzUthuL-6K7jRLM9F5yl4YedWasoT366c align="center")

What we see is that a user sends a request to an orchestrator, which knows which requests to send to Service 1, Service 2, and Service 3. When the orchestrator receives the result of an operation, it responds to the user with it.

The order of the requests (or events) depends on the business process, so the orchestrator does not force you to make calls in parallel or sequentially. As such, it does not force you to use HTTP, gRPC, or a message broker. All of these are implementation details that depend entirely on you.

Let's imagine we have the following flow to buy some products:

![](https://cdn.hashnode.com/uploads/covers/64c01f633d0d5beec2f4cb9f/b0371a19-cd21-4b31-8066-828d3d45a35f.svg align="center")

![](https://docs.2getherme.nl/api/files.get?sig=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJrZXkiOiJ1cGxvYWRzLzQ0NmZkYTIzLTM5MmItNGZmOS1hYjE3LWQ3NjJjOGI3YjI0Zi80NzdmMzVlZi1iMmY3LTQwYjUtOTQ0MS1hMWU4NDM3YzE0NTYvZGlhZ3JhbS5zdmciLCJ0eXBlIjoiYXR0YWNobWVudCIsImlhdCI6MTc4NDgzMDcxMSwiZXhwIjoxNzg1NDM1NTExfQ.8OnbeT5httrURF3_ZrAmQRo5l7RrlbjAyAt7vw6xxwA align="center")

The flow is straightforward:

*   Creating the order is the first step. Until we create an order, we cannot book inventory in a warehouse. If the order is not created, we cannot start the process.
    
*   The order is created, so we should book inventory for it. If, for some reason, this cannot be done, we cannot proceed to the next step.
    
*   The inventory is booked, so we can send a confirmation email to the customer.
    
*   In the last step, we create the delivery.
    

This process is quite linear: we send a request, wait for the result, and trigger the next step. In reality, not all processes are like this. It might be that at some stage, we need to run a few processes in parallel, wait until all of them finish, combine the results, and decide what to do next.

# Direct-Call Orchestrator

This type of orchestrator makes direct calls to known services. The most common way to implement it is using REST or gRPC. Let's first look at the code snippet below:

```csharp
public async Task<OrderResult> ProcessOrderAsync(Order order)
{
    try
    {
        var orderResult = await _orderClient.CreateOrderAsync(order);
        var paymentResult = await _paymentClient.ProcessPaymentAsync(order.Payment);
        var emailResult = await _emailClient.SendOrderEmailAsync(order);
        var inventoryResult = await _inventoryClient.ReserveItemsAsync(order.Items);
        
        return new OrderResult { Success = true };
    }
    catch (Exception ex)
    {
        return new OrderResult { Success = false };
    }
}
```

This simple method outlines the main idea of a direct-call orchestrator. Here, we have several clients for different services, and we simply call them one after another.

**Pros:**

*   It is the simplest implementation, which is easy to extend.
    
*   You have complete control over the process, making it easy to debug and run.
    

Cons:

*   Tightly coupled design. The orchestrator can only work when all dependencies are healthy. Imagine a situation where one service is down, and the entire process goes down as well.
    
*   There is no built-in retry mechanism, so all clients likely need to be adjusted to use retry or circuit breaker libraries.
    

# Event-Driven Orchestrator

This type of orchestrator relies on events rather than direct calls to services. Let's look at a possible implementation of this approach using a background service and Azure Service Bus:

```csharp
public class OrderOrchestratorService : BackgroundService
{
    private readonly ServiceBusProcessor _eventProcessor;
    private readonly ServiceBusSender _commandSender;
    private readonly ILogger<OrderOrchestratorService> _logger;
    private readonly ConcurrentDictionary<Guid, TaskCompletionSource<object>> _pendingSteps = new();

    public OrderOrchestratorService(
        ServiceBusClient serviceBusClient,
        ILogger<OrderOrchestratorService> logger)
    {
        _eventProcessor = serviceBusClient.CreateProcessor("orchestrator-events", "orchestrator-sub");
        _commandSender = serviceBusClient.CreateSender("orchestrator-commands");
        _logger = logger;

        _eventProcessor.ProcessMessageAsync += OnEventReceived;
        _eventProcessor.ProcessErrorAsync += OnError;
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        await _eventProcessor.StartProcessingAsync(stoppingToken);
        _logger.LogInformation("OrderOrchestrator started");

        try
        {
            await Task.Delay(Timeout.Infinite, stoppingToken);
        }
        catch (OperationCanceledException)
        {
        }

        await _eventProcessor.StopProcessingAsync();
    }

    public override async Task StopAsync(CancellationToken cancellationToken)
    {
        await _commandSender.DisposeAsync();
        await _eventProcessor.DisposeAsync();
        await base.StopAsync(cancellationToken);
    }
    
    // Remaining code
}
```

Events are the fuel of this orchestrator. With the start of the background process, we create an event processor that listens to the `orchestrator-events` topic. This topic receives all status events about the process. When it receives a message, the event handler `OnEventReceived` is called.

Additionally, we create an Azure Service Bus sender for the `orchestrator-commands` topic. Services subscribe to this topic to receive commands to perform their part of the process.

Let's look at how the `OnEventReceived` method works:

```csharp
private async Task OnEventReceived(ProcessMessageEventArgs args)
{
    var message = args.Message;
    var correlationId = Guid.Parse(message.CorrelationId);

    if (message.Subject == "OrderPlaced")
    {
        var orderPlaced = JsonSerializer.Deserialize<OrderPlaced>(message.Body.ToStream())!;

        _ = ProcessOrderAsync(orderPlaced, args.CancellationToken);
    }
    else
    {
        if (_pendingSteps.TryRemove(correlationId, out var tcs))
            tcs.TrySetResult(DeserializeResult(message));
    }

    await args.CompleteMessageAsync(message);
}
```

The orchestrator starts with an `OrderPlaced` event, which calls the `ProcessOrderAsync` method. All other messages are steps of the orchestration process, registered in `_pendingSteps`. When the result of a step is received, it is removed from the pending list.

Let's look at the `ProcessOrderAsync` method:

```csharp
private async Task ProcessOrderAsync(OrderPlaced order, CancellationToken ct)
{
    try
    {
        var bookResult = await SendAndWaitAsync<BookInventoryResult>(
            new BookInventoryCommand
            {
                CorrelationId = order.CorrelationId,
                OrderId = order.OrderId,
                Product = order.Product,
                Amount = order.Amount
            },
            "BookInventory",
            order.CorrelationId);

        if (!bookResult.Success)
        {
            return;
        }

        await SendAndWaitAsync<SendEmailResult>(
            new SendEmailCommand
            {
                CorrelationId = order.CorrelationId,
                OrderId = order.OrderId,
                CustomerEmail = order.CustomerEmail
            },
            "SendEmail",
            order.CorrelationId);

        await SendAndWaitAsync<CreateDeliveryResult>(
            new CreateDeliveryCommand
            {
                CorrelationId = order.CorrelationId,
                OrderId = order.OrderId
            },
            "CreateDelivery",
            order.CorrelationId); 
    }
    catch (Exception ex)
    {
        _logger.LogError(ex,
            "[{CorrelationId}] Orchestrator: Order orchestration failed",
            order.CorrelationId);
    }
}
```

The main idea of this code is to send an event for a step and wait until we have a result. This is implemented in the `SendAndWaitAsync` method. The orchestrator stops processing if the inventory cannot be booked. At the same time, we do not check the results for `SendEmail` and `CreateDelivery`.

```csharp
private async Task<T> SendAndWaitAsync<T>(object command, string commandType, Guid correlationId)
{
    var tcs = new TaskCompletionSource<object>(TaskCreationOptions.RunContinuationsAsynchronously);
    _pendingSteps[correlationId] = tcs;

    var message = new ServiceBusMessage(JsonSerializer.SerializeToUtf8Bytes(command))
    {
        Subject = commandType,
        CorrelationId = correlationId.ToString()
    };
    await _commandSender.SendMessageAsync(message);

    var result = await tcs.Task;
    return (T)result;
}
```

The `SendAndWaitAsync` method uses the `TaskCompletionSource` class to wait for the command to complete. This code looks like common async/await, but in reality, its execution relies on `OnEventReceived`, which is executed each time a new message is published.

The complete source code can be found on [GitHub](https://github.com/MasDenBy/blog-samples/tree/main/orchestrator-example).

**Pros:**

*   The design is more decentralized compared to the direct-call orchestrator. It does not rely on specific services—only on messages. It does not matter who or how processes commands, as long as they do their job and respond in the correct format.
    

**Cons:**

*   Difficult to implement, as we need to use a wait mechanism for steps. Additionally, we need to store data for the orchestrator somewhere. In my example, I store it in memory, but memory has its limits. If we need to update the orchestrator's code and restart it, the data in memory will be lost. Thus persistence mechanism is necessary.
    

# Problems and Considerations

## Single Point of Failure

The orchestrator becomes a single point of failure for the entire workflow. If it fails, the entire business process can stall. Thus, it requires high availability and fault tolerance.

## Possible Scalability Bottleneck

For high-load applications that require processing a huge number of events, the orchestrator might become a bottleneck. It is important to consider possible scaling solutions.

## Additional complexity

The orchestrator introduces additional complexity into the business workflow. It needs to manage events, combine responses, and handle retries for direct calls. Additionally, it requires infrastructure to store state.

## How to Implement

We have already seen two possible implementations of this pattern and quickly outlined their pros and cons. In reality, the way you implement it mostly aligns with your business needs and the infrastructure you use. In some situations, simple direct calls may be better than a complicated event-driven approach, or you may need to combine both.

Consider whether you are going to implement the infrastructure yourself or use ready-to-use solutions. If you do not have time to fix the infrastructure layer of your orchestrator and just want to focus on business logic, I recommend looking at existing implementations.

### Azure Durable Functions

Azure Durable Functions is an extension of Azure Functions that enables stateful, long-running workflows in a serverless environment. With it, developers can write orchestration logic to coordinate multiple steps, handle retries, manage state, and recover from failures without managing the infrastructure.

Since Durable Functions is an extension of Azure Functions, you have all the features, such as seamless integration with Azure services and scalability out of the box. If you use Azure and Azure Functions, it might be the first choice.

### MassTransit

[MassTransit](https://masstransit.massient.com/) is a distributed application framework for building message-based systems that are reliable under load, testable during development, and observable in production. It supports different persistence mechanisms, from SQL databases to cloud storage like Amazon S3 or Azure Storage.

The documentation is full of examples on how to configure state, monitoring, or create a job. It might not be the easiest library to use, but the number of examples and the framework's popularity can help. MassTransit does not have a ready-to-use implementation of an orchestrator, but with its help, it can be created as a separate service or a saga. Thus, you can use it with any cloud provider or on-premise.

### **Elsa Workflows**

[Elsa Workflows](https://docs.elsaworkflows.io/) another interesting, lightweight open-source project which allows running workflows in .NET. It supports both code-first and designer-first approaches, allowing workflows to be defined in C# or using a visual designer (Elsa Studio).

Elsa Workflows is an orchestrator by design. So, you do not need to think about how to define orchestrator logic - you just define orchestrator steps. It supports many persistence providers, event-driven flows, retries, and event handling out of the box. It might be a good choice if you are looking for a ready-to-use orchestrator.

# When to Use

## Complex or Long Workflows

The orchestrator is ideal when you have complex workflows or long-running processes that might take hours or days. Think about processes where you need to execute steps in a specific order or with additional conditional logic.

## Strong Consistency

You need to enforce strong consistency for business rules, such as:

*   "The payment must succeed before shipping."
    
*   "The user should be created before assigning a subscription."
    

Using an orchestrator helps mitigate issues when some processes do not finish while others have already started.

# When Not to Use

## Simple, Short-Lived Workflows

If your workflow is trivial and consists of only a few steps, you likely do not need an orchestrator. Consider using [Choreography](https://iharmaiseyeu.net/choreography-pattern-in-microservices) instead.

## Non-Critical Processes

If the workflow can tolerate failures without requiring compensation or retries, you do not need an orchestrator. Additionally, retries can sometimes be achieved without an orchestrator by using message brokers.

## Performance-Critical Workflows

The orchestrator adds latency overhead due to orchestration and persistence logic. For performance-critical applications, this might outweigh the benefits of orchestration. In such cases, consider other approaches like [Choreography](https://iharmaiseyeu.net/choreography-pattern-in-microservices) or direct calls.

# Summary

The Orchestration pattern is a useful tool when your workflows need ordering and guidance. However, it adds additional latency and state persistence overhead. It is always good to examine your process from different angles and understand whether you need strong consistency and if you are comfortable with the single point of failure that comes with implementing this pattern.

# Links

[**Durable Orchestrations Overview - Azure | Microsoft Learn**](https://learn.microsoft.com/en-us/azure/durable-task/common/durable-task-orchestrations)

[MassTransit](https://github.com/MassTransit/MassTransit)

[Elsa Workflows](https://docs.elsaworkflows.io/)

[The Source Code on GitHub](https://github.com/MasDenBy/blog-samples/tree/main/orchestrator-example)

Cover image by [Uwe Conrad](https://pixabay.com/users/scratsmacker-16310259/?utm_source=link-attribution&utm_medium=referral&utm_campaign=image&utm_content=5468470) from [Pixabay](https://pixabay.com//?utm_source=link-attribution&utm_medium=referral&utm_campaign=image&utm_content=5468470)