Use Testcontainers for running Azurite in Azure DevOps pipeline

Introduction

In our solutions, we use storage accounts quite often. They are easy and inexpensive, and for many situations, they are sufficient, eliminating the need to pay for expensive alternatives.

Azurite is a storage emulator of storage accounts for local development, allowing you to work with blobs, queues, and tables on your machine without incurring any costs. When necessary, you can simply switch to using an Azure Storage account in the cloud. It's as easy as a piece of cake, right?

For local development, it certainly is. But what if we need to write integration tests that require communication with a storage account? I see a few possible solutions to achieve that.

Use real instances of Azure storage account

At first glance, this seems like a good idea. After all, we are already using an Azure storage account, so why not leverage it for testing purposes? Indeed, we can, but if multiple developers share a single instance, they may inadvertently interfere with each other. Therefore, each developer should utilize their dedicated storage account. However, this approach is not without its drawbacks.

Firstly, we require a minimum of two storage accounts—one for development and another for testing.

Secondly, it's essential to ensure that the storage account is empty before executing tests, necessitating regular cleanup.

Thirdly, if you happen to be working in an environment with an unreliable internet connection, running tests can become problematic.

Fourthly, in the context of a CI pipeline, providing access to multiple storage accounts may not be feasible. While it's possible to create a single account for testing, it still requires cleanup after tests and careful coordination to ensure that only one pipeline runs at a time. This can be cumbersome for larger teams where multiple developers need to run tests frequently.

Utilizing the Locally Installed Version of Azurite

Azurite is an emulator of a storage account that can be installed on a local environment. While this approach addresses some issues related to an unstable internet connection, it is not without its challenges.

Azurite is well-suited for local development, and I strongly encourage its use for that purpose. However, when it comes to running tests, we encounter the same challenge of needing a separate, clean testing environment.

Leveraging Testcontainers

Testcontainers a powerful and versatile library that simplifies the process of containerized testing. It allows you to spin up Docker containers for databases, message brokers, and other dependencies your application relies on. It offers a set of packages designed to streamline the provisioning of resources for testing.

In simple terms, it operates as follows: when we require a service, an instance of it is provisioned within a Docker container. We can then interact with it just like a regular instance. Once the work is complete (i.e., all tests have been executed), the container is automatically destroyed.

With Testcontainers, each container instance is created as a pristine environment, eliminating the need for manual cleanup. Since it operates on a local machine, each developer can be confident that their tests run in a completely isolated environment.

Internet connectivity is only necessary during the initial setup when Docker may need to download images if they are not already present in the local environment.

As the name suggests, Testcontainers is primarily intended for use in testing. Of course, no one can stop you from using them for development tasks as well.

Sample application

To present how to use Testcontainers I have written a minimal API with one endpoint that downloads a file from an Azure blob and displays it to the user. The source code is below:

using Azure.Storage.Blobs;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddSingleton(x =>
{
    var connectionString = "<your-connection-string>";
    return new BlobServiceClient(connectionString);
});

var app = builder.Build();

app.MapGet("/api/file", async (BlobServiceClient blobServiceClient) =>
{
    var containerClient = blobServiceClient.GetBlobContainerClient("data");
    var blobClient = containerClient.GetBlobClient("file.txt");

    var response = await blobClient.DownloadContentAsync();
    return Results.Text(response.Value.Content);
});

app.Run();

Since I don't have Azurite installed on my machine, I use a Docker image with the following command:

docker run -p 10000:10000 mcr.microsoft.com/azure-storage/azurite azurite-blob --blobHost 0.0.0.0

Once the emulator is completely initialized, I create the container with the name data and upload a file named file.txt with the content Content of the file.

Running the API and executing the endpoint should return the text Content of the file. We can run it locally to verify that everything is working, and indeed, it does. However, we would like to have a more reliable and faster mechanism to validate this functionality. Therefore, let's add an integration test for it.

Adding integration tests

Well-written tests should be as isolated as possible. For our endpoint, we cannot use the same storage account as for development or production environments. Thus, we can highlight that our test should be able to:

Create a storage account and a container.
Upload a file into the container.
Delete the storage account.

To address the first and last points, we will use Testcontainers.

What we need first is to add the Testcontainers.Azurite package to our test project. Many packages are available for different services, and the full list can be observed through this link.

dotnet add package Testcontainers.Azurite

In our test we upload the file with some text content, call the endpoint to return the content of the uploaded file, and validate that it is the same.

public class ProgramTests : IClassFixture<ApiWebApplicationFactory>
{
    private readonly HttpClient httpClient;
    private readonly BlobHelper blobHelper;

    public ProgramTests(ApiWebApplicationFactory factory)
    {
        this.httpClient = factory.CreateClient();
        this.blobHelper = new BlobHelper(factory.Services);
    }

    [Fact]
    public async Task Get_ShouldResponseWithTextFileContent()
    {
        // Arrange
        const string content = "The content of the file for test";
        await this.blobHelper.UploadTextFileAsync("file.txt", content);

        // Act
        var response = await this.httpClient.GetAsync("/api/file");
        response.EnsureSuccessStatusCode();

        var actualContent = await response.Content.ReadAsStringAsync();

        // Assert
        Assert.Equal(content, actualContent);
    }
}

BlobHelper is a helper class for uploading the file. The content of it is available in the source code.

ApiWebApplicationFactory is our custom implementation of WebApplicationFactory to configure our integration tests.

public class ApiWebApplicationFactory : WebApplicationFactory<Program>, IAsyncLifetime
{
    private readonly AzuriteTestContainer azuriteTestContainer;

    public ApiWebApplicationFactory()
    {
         this.azuriteTestContainer = new AzuriteTestContainer();
    }

    public async Task InitializeAsync()
    {
        await this.azuriteTestContainer.StartAsync();
    }

    protected override void ConfigureWebHost(IWebHostBuilder builder)
    {
        builder.ConfigureServices(services =>
        {
            services.AddSingleton(new BlobServiceClient(azuriteTestContainer.ConnectionString));
        });
    }

    async Task IAsyncLifetime.DisposeAsync()
    {
        await this.azuriteTestContainer.DisposeAsync();
    }
}

Here, we create AzuriteTestContainer (see the source code of this object below) and perform the essential parts: start the container and dispose of it.

using Testcontainers.Azurite;

internal class AzuriteTestContainer
{
    private readonly AzuriteContainer container;

    public AzuriteTestContainer()
    {
        this.container = new AzuriteBuilder()
            .WithImage("mcr.microsoft.com/azure-storage/azurite:latest")
            .Build();
    }

    public string ConnectionString => this.container.GetConnectionString();

    public Task StartAsync() => this.container.StartAsync();
    public ValueTask DisposeAsync() => this.container.DisposeAsync();
}

The AzuriteTestContainer class serves as a wrapper around AzuriteContainer and AzuriteBuilder from the Testcontainers.Azurite package. This object is responsible for creating a Docker container with the specified image and disposing of it.

Running our test will automatically provision Azurite in a Docker container.

In addition to the container, we expect to see another one named testcontainers/ryuk, which serves as a helper for cleaning all resources (containers, networks, volumes) created during the test run.

The mcr.microsoft.com/azure-storage/azurite image represents the Azurite container. It's important to note that the ports may differ, so if you run some tests in parallel and several Azurite containers are created, each will have its ports. Therefore, you don't need to write any lines of code for that.

In our Azure DevOps pipeline, we can add a task to run tests like that.

- task: DotNetCoreCLI@2
  displayName: test
  inputs:
    command: test
    projects: $(solution)
    arguments: --configuration Release --logger trx --collect:"XPlat Code Coverage"

At this point, our integration test will pass in the pipeline. Examining the console input, we can see that the Docker container was automatically provisioned and de-provisioned automatically. Cool, isn't it?

...
[testcontainers.org 00:00:01.64] Docker image testcontainers/ryuk:0.5.1 created
[testcontainers.org 00:00:01.71] Docker container 0d6de9a94a02 created
[testcontainers.org 00:00:01.77] Start Docker container 0d6de9a94a02
[testcontainers.org 00:00:03.13] Wait for Docker container 0d6de9a94a02 to complete readiness checks
[testcontainers.org 00:00:03.13] Docker container 0d6de9a94a02 ready
[testcontainers.org 00:00:03.15] Docker registry credential mcr.microsoft.com not found
[testcontainers.org 00:00:08.30] Docker image mcr.microsoft.com/azure-storage/azurite:latest created
[testcontainers.org 00:00:08.33] Docker container 21715eda45f1 created
[testcontainers.org 00:00:08.33] Start Docker container 21715eda45f1
[testcontainers.org 00:00:08.66] Wait for Docker container 21715eda45f1 to complete readiness checks
[testcontainers.org 00:00:08.66] Docker container 21715eda45f1 ready
[testcontainers.org 00:00:15.35] Delete Docker container 21715eda45f1
Results File: /home/vsts/work/_temp/_fv-az429-553_2023-10-01_10_54_49.trx

Passed!  - Failed:     0, Passed:     1, Skipped:     0, Total:     1, Duration: < 1 ms - AzuriteTestContainers.IntegrationTests.dll (net7.0)
...

Move to managed identity

Currently, we are using a connection string to establish a connection with the Azure storage account. However, this is not the best approach, and Microsoft recommends using Managed Identity instead. Let's update our code to align with this approach.

We need to remove the connection string and use blob URI and DefaultAzureCredential instead. Thus instantiation of BlobServiceClient should look like this.

builder.Services.AddSingleton(x =>
{
    return new BlobServiceClient(new Uri("http://127.0.0.1:10000/devstoreaccount1"), new DefaultAzureCredential());
});

When running the application, you might encounter the following error: System.ArgumentException: Cannot use TokenCredential without HTTPS.

Managed identity cannot be used when the service is accessible via HTTP as Azurite runs by default, so we need to address this issue. The Azurite GitHub page provides information on how to enable HTTPS by providing a certificate.

To generate the certificate, we'll use the utility mkcert, which allows us to quickly create development certificates with just a few commands:

mkcert -install
mkcert 127.0.0.1

The first command creates a local Certificate Authority (CA) and installs it in the system's trust store. The second command generates our development certificate for the local system.

Now, we can run our Docker container using the following command:

docker run -p 10000:10000 -v c:/certs/:/workspace  mcr.microsoft.com/azure-storage/azurite azurite --blobHost 0.0.0.0 --cert /workspace/127.0.0.1.pem --key /workspace/127.0.0.1-key.pem --oauth basic

A crucial step here is to map (-v c:/certs/:/workspace) the folder containing the certificates to Docker.

With this setup, we are ready to change the blob URL from http://127.0.0.1:10000/devstoreaccount1 to https://127.0.0.1:10000/devstoreaccount1.

After migrating our app to use managed identity, we should do the same with our tests. While it's possible to keep tests running with a connection string, it's not ideal for ensuring 100% confidence in our unit tests. To achieve this, we need to be as closely aligned with our application code as possible.

To enable Azurite to run over HTTPS, we need to use certificates. Since we've already generated development certificates, we just need to make them accessible for tests. To do this, create a folder named certs in the test project and copy the certificate files into it.

Next, modify the AzuriteTestContainer class to work with certificates and return the correct URLs of services instead of the connection string. Below is the source code for this class:

internal class AzuriteTestContainer
{
    private static readonly Regex BlobUriRegex = new(@"BlobEndpoint=([A-Za-z0-9:\\.\/]*)", RegexOptions.Compiled);

    private readonly AzuriteContainer container;
    private Uri blobUri = null!;

    public AzuriteTestContainer()
    {
        this.container = new AzuriteBuilder()
            .WithImage("mcr.microsoft.com/azure-storage/azurite:latest")
            .WithResourceMapping(new DirectoryInfo("./certs"), "/workspace/")
            .WithCommand("--cert", "/workspace/127.0.0.1.pem", "--key", "/workspace/127.0.0.1-key.pem", "--oauth", "basic")
            .Build();
    }

    public Uri BlobUri => this.blobUri ??= this.GetServiceUri(BlobUriRegex);

    public Task StartAsync() => this.container.StartAsync();
    public ValueTask DisposeAsync() => this.container.DisposeAsync();

    private Uri GetServiceUri(Regex regex)
    {
        var connectionString = this.container.GetConnectionString();

        var matchResult = regex.Match(connectionString);

        var uri = matchResult.Success 
            ? new Uri(matchResult.Groups[1].Value)
            : throw new InvalidOperationException("Cannot retrieve uri from connection string");

        var uriBuilder = new UriBuilder(uri)
        {
            Scheme = Uri.UriSchemeHttps
        };

        return uriBuilder.Uri;
    }
}

Let's review these changes:

WithResourceMapping maps the directory from our local system to the directory in the container, making our certificates available to Azurite inside the container instance.
WithCommand allows us to run a command inside the container during startup. We use the same command that we used to run the Docker container on the local system.

Since we no longer use a connection string, we need the real addresses of the services (in our example, we use only the blob service, but it's easy to extend for tables and queues). Unfortunately, AzuriteContainer does not return such data by default, but we can easily retrieve it from a connection string. We use GetServiceUri and BlobUriRegex for that.

There is a small change in ApiWebApplicationFactory to use the blob URI instead of the connection string:

services.AddSingleton(new BlobServiceClient(azuriteTestContainer.BlobUri, new DefaultAzureCredential()));

After these changes, our application and tests should work as before in the local environment. However, if you run them on CI, they might fail due to missing certificates. To address this issue, make a few small changes in your pipeline:

variables:
  solution: 'azurite-testcontainers/AzuriteTestContainers.sln'
  azureSubscription: 'My Test Subscription'

# Others tasks omitted

- task: CmdLine@2
  displayName: install certificate
  inputs:
    workingDirectory: $(Build.SourcesDirectory)
    script: |
      sudo apt install libnss3-tools -y
      curl -JLO "https://dl.filippo.io/mkcert/latest?for=linux/amd64"
      chmod +x mkcert-v*-linux-amd64
      sudo cp mkcert-v*-linux-amd64 /usr/local/bin/mkcert
      mkcert -install
      mkcert -cert-file ./azurite-testcontainers/AzuriteTestContainers.IntegrationTests/certs/127.0.0.1.pem -key-file ./azurite-testcontainers/AzuriteTestContainers.IntegrationTests/certs/127.0.0.1-key.pem 127.0.0.1

- task: AzureCLI@2
  displayName: test
  inputs:
    azureSubscription: $(azureSubscription)
    scriptType: pscore
    scriptLocation: inlineScript
    inlineScript: |
      dotnet test $(solution) /p:CollectCoverage=true /p:CoverletOutputFormat=opencover --collect "Code coverage" --logger trx

# Others tasks omitted

A new step, install certificate, has been added. If you're using a Windows machine, you can find information about the installation process on the GitHub page of the utility.

We can no longer use the DotNetCoreCLI@2 task to run tests because we now use managed identity in our tests, which require running them with an identity. If you deploy your application to Azure, you use a subscription for that purpose, and you can use the same subscription to run tests. To do so, use the AzureCLI@2 task and specify the subscription to use.

Summary

Testing is an integral part of our daily routine, and libraries like Testcontainers make it easier. We have demonstrated how simple it is to use Azurite in a container, and our minor changes in the testing infrastructure further validate this approach.