Running integration tests inside Service Fabric

I’m currently diving into Service Fabric. One of the difficult things that you’ll be faced with when you write code against the Service Fabric APIs is how you can integration test those components. Infrastructure like the reliable state manager is only available inside Service Fabric. So somehow you need to run the tests that are testing the components that use infrastructure like reliable state manager inside the cluster. In this post, I show you an approach which allows running integration tests written in NUnit inside Service Fabric. The full version of the code can be found in my ServiceFabric.Testing GitHub repository.

The idea is simple:

1. Create an application with a stateful service that hosts the integration tests
2. Run the tests inside the stateful service hosted in the cluster
3. Report the progress and the results back to the build server (I’ll be using TeamCity as an example)

Let’s build this thing. For simplicity reasons, I show only how you could get access to the reliable state manager. All tests that need access to the reliable state manager could inherit from StatefulServiceContextAwareBase. The base class extracts the reliable state manager from the TestContext provided by NUnit.

public abstract class StatefulServiceContextAwareBase
{
    public IReliableStateManager StateManager { get; set; } =
        TestContext.CurrentContext.Test.Properties.Get("ReliableStateManager") as IReliableStateManager;
}

But somehow the state manager needs to get into the text context. This magic happens in the RunAsync method of the stateful service that hosts the tests.

ConcurrentQueue<string> output = new ConcurrentQueue<string>();

protected override async Task RunAsync(CancellationToken cancellationToken) {
    
    var runner = new NUnitTestAssemblyRunner(new DefaultTestAssemblyBuilder());
    runner.Load(GetType().Assembly, new Dictionary<string, object>());
    runner.RunAsync(new CompositeListener(
        new ContextAwareTestListener(StateManager),
        new TeamCityEventListener(new TextWriterConcurrenctQueueDecorator(output))), TestFilter.Empty);

    using (cancellationToken.Register(() => runner.StopRun(force: false)))
    {
        // rest omitted
    }
}

As a package dependency, I used NUnitLite. It has everything you need to host tests inside a library, console or like we do inside a ServiceFabric stateful service. The assumption of the sample I show here is that all tests are written in the same assembly that contains the stateful service definition. With that assumption / convention, I can create an NUnitTestAssemblyRunner and load the current assembly as a test assembly into the runner. In the RunAsync method of the test runner, we can provide implementations of the ITestListener interface. Test listeners can report state changes of tests or interact with the tests definitions.  In my case, I used a composite listener who manages a series of listeners. The responsibility of the ContextAwareTestListener is to fill the reliable state manager into the test properties so that the StatefulServiceContextAwareBase can read it again. Let’s see how it looks like

class ContextAwareTestListener : ITestListener {
    private IReliableStateManager statefulStateManager;

    public ContextAwareTestListener(IReliableStateManager stateManager) {
        statefulStateManager = stateManager;
    }

    public void TestStarted(ITest test) {
        test.Properties.Add("ReliableStateManager", statefulStateManager);
    }

    // ...
}

Since I’m running the tests inside TeamCity, I add the TeamCityEventListener defined in the NUnitLite package. The listener adds messages to a TextWriter which can be interpreted by TeamCity to visualise the state and the outcome of test runs. In my example, I used a ConcurrentQueue<string> to insert the lines and read it again. So I wrote a StringWriter adapter which forwards WriteLines into the concurrent queue.

class TextWriterConcurrenctQueueDecorator : StringWriter {
    private ConcurrentQueue<string> output;

    public TextWriterConcurrenctQueueDecorator(ConcurrentQueue<string> output) {
        this.output = output;
    }

    public override void WriteLine(string format, object arg0) {
        output.Enqueue(string.Format(format, arg0));
    }

    // ...
}

To get the test output from inside the cluster to TeamCity I’ve created a communication listener which exposes a web listener inside ServiceFabric.

private async Task ProcessInternalRequest(HttpListenerContext context, CancellationToken cancelRequest) {
    try
    {
        if (output.Count == 0)
        {
            return;
        }

        using (HttpListenerResponse response = context.Response)
        using(var streamWriter = new StreamWriter(response.OutputStream))
        {
            response.ContentType = "text/plain";
            string line;
            while (output.TryDequeue(out line))
            {
                await streamWriter.WriteLineAsync(line).ConfigureAwait(false);
            }
            await streamWriter.FlushAsync().ConfigureAwait(false);
            streamWriter.Close();
            response.Close();
        }
    }
    catch (Exception)
    {
        // stream closed etc.
    }
}

Everytime a web request comes in the content of the concurrent queue is written to the output stream. For the endpoint to properly work we need to declare it in the ServiceManifest.xml like

  <Resources>
    <Endpoints>
      <Endpoint Name="Web" Protocol="http" UriScheme="http" Port="8089" />
...
    </Endpoints>
  </Resources>

The client code becomes dead simple since ServiceFabric has a reverse proxy built-in we can call to the proxy URI according to the following schema http://{fqdn}:{port}/{Application}/{Service}/{Endpoint}. Since we are running on localhost with TestApplication and TestRunner as service name the URI should be http://localhost:19081/TestApplication/TestRunner/Web.

var httpClient = new HttpClient();

var content = await httpClient.GetAsync("http://localhost:19081/TestApplication/TestRunner/Web");
var stringContent = await content.Content.ReadAsStringAsync();
if (!string.IsNullOrEmpty(stringContent))
{
    Console.Write(stringContent);
}

The code above could be executed in a loop until the test suite end is seen in the http content.

The final part is to setup TeamCity build steps

And here is how the report looks like with live updates.

I hope that helps to get started.

My friend Szymon Kulec came up with an even more elegant way that also works from inside Visual Studio or the Resharper, test runner. Make sure you follow his blog!