Introduction
In my previous article about Choreography, I outlined that it does not suit situations with complicated workflows or where strong consistency is required. So, it was logical to discuss Orchestration, the pattern that helps solve - if not all - many of the issues of Choreography.
What Is the Orchestration Pattern?
If we compare Choreography to dancers following each other, we can imagine Orchestration as an orchestra following a conductor who knows where and when each musician should start their part. The main idea behind this pattern is to have one service that understands the business process and the order in which operations should be called.
When we talk about the business process, we do not mean that the orchestrator should implement the business logic. On the contrary, it should not implement any business logic—only call the services that implement parts of the business logic, check the results, and decide whether the process should continue.
The diagram below outlines a simple orchestration process.
What we see is that a user sends a request to an orchestrator, which knows which requests to send to Service 1, Service 2, and Service 3. When the orchestrator receives the result of an operation, it responds to the user with it.
The order of the requests (or events) depends on the business process, so the orchestrator does not force you to make calls in parallel or sequentially. As such, it does not force you to use HTTP, gRPC, or a message broker. All of these are implementation details that depend entirely on you.
Let's imagine we have the following flow to buy some products:
The flow is straightforward:
Creating the order is the first step. Until we create an order, we cannot book inventory in a warehouse. If the order is not created, we cannot start the process.
The order is created, so we should book inventory for it. If, for some reason, this cannot be done, we cannot proceed to the next step.
The inventory is booked, so we can send a confirmation email to the customer.
In the last step, we create the delivery.
This process is quite linear: we send a request, wait for the result, and trigger the next step. In reality, not all processes are like this. It might be that at some stage, we need to run a few processes in parallel, wait until all of them finish, combine the results, and decide what to do next.
Direct-Call Orchestrator
This type of orchestrator makes direct calls to known services. The most common way to implement it is using REST or gRPC. Let's first look at the code snippet below:
public async Task<OrderResult> ProcessOrderAsync(Order order)
{
try
{
var orderResult = await _orderClient.CreateOrderAsync(order);
var paymentResult = await _paymentClient.ProcessPaymentAsync(order.Payment);
var emailResult = await _emailClient.SendOrderEmailAsync(order);
var inventoryResult = await _inventoryClient.ReserveItemsAsync(order.Items);
return new OrderResult { Success = true };
}
catch (Exception ex)
{
return new OrderResult { Success = false };
}
}
This simple method outlines the main idea of a direct-call orchestrator. Here, we have several clients for different services, and we simply call them one after another.
Pros:
It is the simplest implementation, which is easy to extend.
You have complete control over the process, making it easy to debug and run.
Cons:
Tightly coupled design. The orchestrator can only work when all dependencies are healthy. Imagine a situation where one service is down, and the entire process goes down as well.
There is no built-in retry mechanism, so all clients likely need to be adjusted to use retry or circuit breaker libraries.
Event-Driven Orchestrator
This type of orchestrator relies on events rather than direct calls to services. Let's look at a possible implementation of this approach using a background service and Azure Service Bus:
public class OrderOrchestratorService : BackgroundService
{
private readonly ServiceBusProcessor _eventProcessor;
private readonly ServiceBusSender _commandSender;
private readonly ILogger<OrderOrchestratorService> _logger;
private readonly ConcurrentDictionary<Guid, TaskCompletionSource<object>> _pendingSteps = new();
public OrderOrchestratorService(
ServiceBusClient serviceBusClient,
ILogger<OrderOrchestratorService> logger)
{
_eventProcessor = serviceBusClient.CreateProcessor("orchestrator-events", "orchestrator-sub");
_commandSender = serviceBusClient.CreateSender("orchestrator-commands");
_logger = logger;
_eventProcessor.ProcessMessageAsync += OnEventReceived;
_eventProcessor.ProcessErrorAsync += OnError;
}
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
await _eventProcessor.StartProcessingAsync(stoppingToken);
_logger.LogInformation("OrderOrchestrator started");
try
{
await Task.Delay(Timeout.Infinite, stoppingToken);
}
catch (OperationCanceledException)
{
}
await _eventProcessor.StopProcessingAsync();
}
public override async Task StopAsync(CancellationToken cancellationToken)
{
await _commandSender.DisposeAsync();
await _eventProcessor.DisposeAsync();
await base.StopAsync(cancellationToken);
}
// Remaining code
}
Events are the fuel of this orchestrator. With the start of the background process, we create an event processor that listens to the orchestrator-events topic. This topic receives all status events about the process. When it receives a message, the event handler OnEventReceived is called.
Additionally, we create an Azure Service Bus sender for the orchestrator-commands topic. Services subscribe to this topic to receive commands to perform their part of the process.
Let's look at how the OnEventReceived method works:
private async Task OnEventReceived(ProcessMessageEventArgs args)
{
var message = args.Message;
var correlationId = Guid.Parse(message.CorrelationId);
if (message.Subject == "OrderPlaced")
{
var orderPlaced = JsonSerializer.Deserialize<OrderPlaced>(message.Body.ToStream())!;
_ = ProcessOrderAsync(orderPlaced, args.CancellationToken);
}
else
{
if (_pendingSteps.TryRemove(correlationId, out var tcs))
tcs.TrySetResult(DeserializeResult(message));
}
await args.CompleteMessageAsync(message);
}
The orchestrator starts with an OrderPlaced event, which calls the ProcessOrderAsync method. All other messages are steps of the orchestration process, registered in _pendingSteps. When the result of a step is received, it is removed from the pending list.
Let's look at the ProcessOrderAsync method:
private async Task ProcessOrderAsync(OrderPlaced order, CancellationToken ct)
{
try
{
var bookResult = await SendAndWaitAsync<BookInventoryResult>(
new BookInventoryCommand
{
CorrelationId = order.CorrelationId,
OrderId = order.OrderId,
Product = order.Product,
Amount = order.Amount
},
"BookInventory",
order.CorrelationId);
if (!bookResult.Success)
{
return;
}
await SendAndWaitAsync<SendEmailResult>(
new SendEmailCommand
{
CorrelationId = order.CorrelationId,
OrderId = order.OrderId,
CustomerEmail = order.CustomerEmail
},
"SendEmail",
order.CorrelationId);
await SendAndWaitAsync<CreateDeliveryResult>(
new CreateDeliveryCommand
{
CorrelationId = order.CorrelationId,
OrderId = order.OrderId
},
"CreateDelivery",
order.CorrelationId);
}
catch (Exception ex)
{
_logger.LogError(ex,
"[{CorrelationId}] Orchestrator: Order orchestration failed",
order.CorrelationId);
}
}
The main idea of this code is to send an event for a step and wait until we have a result. This is implemented in the SendAndWaitAsync method. The orchestrator stops processing if the inventory cannot be booked. At the same time, we do not check the results for SendEmail and CreateDelivery.
private async Task<T> SendAndWaitAsync<T>(object command, string commandType, Guid correlationId)
{
var tcs = new TaskCompletionSource<object>(TaskCreationOptions.RunContinuationsAsynchronously);
_pendingSteps[correlationId] = tcs;
var message = new ServiceBusMessage(JsonSerializer.SerializeToUtf8Bytes(command))
{
Subject = commandType,
CorrelationId = correlationId.ToString()
};
await _commandSender.SendMessageAsync(message);
var result = await tcs.Task;
return (T)result;
}
The SendAndWaitAsync method uses the TaskCompletionSource class to wait for the command to complete. This code looks like common async/await, but in reality, its execution relies on OnEventReceived, which is executed each time a new message is published.
The complete source code can be found on GitHub.
Pros:
- The design is more decentralized compared to the direct-call orchestrator. It does not rely on specific services—only on messages. It does not matter who or how processes commands, as long as they do their job and respond in the correct format.
Cons:
- Difficult to implement, as we need to use a wait mechanism for steps. Additionally, we need to store data for the orchestrator somewhere. In my example, I store it in memory, but memory has its limits. If we need to update the orchestrator's code and restart it, the data in memory will be lost. Thus persistence mechanism is necessary.
Problems and Considerations
Single Point of Failure
The orchestrator becomes a single point of failure for the entire workflow. If it fails, the entire business process can stall. Thus, it requires high availability and fault tolerance.
Possible Scalability Bottleneck
For high-load applications that require processing a huge number of events, the orchestrator might become a bottleneck. It is important to consider possible scaling solutions.
Additional complexity
The orchestrator introduces additional complexity into the business workflow. It needs to manage events, combine responses, and handle retries for direct calls. Additionally, it requires infrastructure to store state.
How to Implement
We have already seen two possible implementations of this pattern and quickly outlined their pros and cons. In reality, the way you implement it mostly aligns with your business needs and the infrastructure you use. In some situations, simple direct calls may be better than a complicated event-driven approach, or you may need to combine both.
Consider whether you are going to implement the infrastructure yourself or use ready-to-use solutions. If you do not have time to fix the infrastructure layer of your orchestrator and just want to focus on business logic, I recommend looking at existing implementations.
Azure Durable Functions
Azure Durable Functions is an extension of Azure Functions that enables stateful, long-running workflows in a serverless environment. With it, developers can write orchestration logic to coordinate multiple steps, handle retries, manage state, and recover from failures without managing the infrastructure.
Since Durable Functions is an extension of Azure Functions, you have all the features, such as seamless integration with Azure services and scalability out of the box. If you use Azure and Azure Functions, it might be the first choice.
MassTransit
MassTransit is a distributed application framework for building message-based systems that are reliable under load, testable during development, and observable in production. It supports different persistence mechanisms, from SQL databases to cloud storage like Amazon S3 or Azure Storage.
The documentation is full of examples on how to configure state, monitoring, or create a job. It might not be the easiest library to use, but the number of examples and the framework's popularity can help. MassTransit does not have a ready-to-use implementation of an orchestrator, but with its help, it can be created as a separate service or a saga. Thus, you can use it with any cloud provider or on-premise.
Elsa Workflows
Elsa Workflows another interesting, lightweight open-source project which allows running workflows in .NET. It supports both code-first and designer-first approaches, allowing workflows to be defined in C# or using a visual designer (Elsa Studio).
Elsa Workflows is an orchestrator by design. So, you do not need to think about how to define orchestrator logic - you just define orchestrator steps. It supports many persistence providers, event-driven flows, retries, and event handling out of the box. It might be a good choice if you are looking for a ready-to-use orchestrator.
When to Use
Complex or Long Workflows
The orchestrator is ideal when you have complex workflows or long-running processes that might take hours or days. Think about processes where you need to execute steps in a specific order or with additional conditional logic.
Strong Consistency
You need to enforce strong consistency for business rules, such as:
Using an orchestrator helps mitigate issues when some processes do not finish while others have already started.
When Not to Use
Simple, Short-Lived Workflows
If your workflow is trivial and consists of only a few steps, you likely do not need an orchestrator. Consider using Choreography instead.
Non-Critical Processes
If the workflow can tolerate failures without requiring compensation or retries, you do not need an orchestrator. Additionally, retries can sometimes be achieved without an orchestrator by using message brokers.
The orchestrator adds latency overhead due to orchestration and persistence logic. For performance-critical applications, this might outweigh the benefits of orchestration. In such cases, consider other approaches like Choreography or direct calls.
Summary
The Orchestration pattern is a useful tool when your workflows need ordering and guidance. However, it adds additional latency and state persistence overhead. It is always good to examine your process from different angles and understand whether you need strong consistency and if you are comfortable with the single point of failure that comes with implementing this pattern.
Links
Durable Orchestrations Overview - Azure | Microsoft Learn
MassTransit
Elsa Workflows
The Source Code on GitHub
Cover image by Uwe Conrad from Pixabay
