Integration Tests in Service Fabric – Client side test cases

In the last post, I showed how it is possible to write test fixture that gets access to the reliable state manager and more that is only available when running on the cluster. In this post, I’ll explain the client side that interacts over remote proxies with the tests that are running on the cluster.

In the post about the test definition I showed that in order to interact with the tests defined in the OrderShippingTests stateful service that is deployed with the OrderShippingTestsApplication I need to define a test class in AllTests project that looks like the following

public class OrderShippingTests : R<OrderShippingTests>
{
}

The R base class has the following responsibility and characteristics

• It has a silly and short name to not clutter the visual tree in the test runner
• It acts as a proxy client to the tests running inside the cluster
• It derives the application name, image store path etc. by convention
• It automatically deploys and removes the test application

Before we dive into the deployment let’s have a look how the tests are reported to the client. The R base class defines a method called underscore. The underscore method is annotated with a TestCaseSource attribute. Every time a new test case arrives the testRunner is called with the test name and then the result is printed on the client. When the test executed in Service Fabric uses output writers they will get automatically intercepted by NUnit and serialized over the remoting interface to the client. If there is output available the HasOutput flag is true and then the string marshaled over is directly written into the console of the client runner. Exceptions are handled similarly. Whenever an exception was raised inside a test running in the cluster the HasException property is set. In that case, the exception is rethrown to the test runner.

[Timeout(600000)]
[TestCaseSource(nameof(GetTestCases))]
public async Task _(string testName) {
    Result result = null;
    try {
        result = await testRunner.Run(testName)
            .ConfigureAwait(false);

        if (result.HasOutput) {
            Console.WriteLine(result.Output);
        }

        if (result.HasException) {
            throw result.Exception;
        }
    }
    finally {
        // diagnostics output
    }
}

Getting the test cases is relatively straight forward. It is a static method that is referenced in the TestCaseSource attribute. Before the test runner can be queried the application has to be deployed to the cluster. This happens in the Setup method and will be discussed in follow-up posts. When the deployment was successful the test runner is queried for all test case names. For all test case names, the infrastructure creates the TestCaseData class that is required for NUnit to represent a test case. The constructor parameter of the TestCaseData structure represents the parameter that will be passed into the test method. For a good representation in the runner I set the TestName property of the TestCaseData.

static IEnumerable<ITestCaseData> GetTestCases() {
    string[] enumerable;
    try {
        SetUp().GetAwaiter().GetResult();

        enumerable = testRunner.Tests().GetAwaiter().GetResult();
    }
    catch (Exception) {
        TearDown().GetAwaiter().GetResult();
        throw;
    }

    foreach (var test in enumerable) {
        var testCaseData = new TestCaseData(test)
        {
            TestName = test
        };
        yield return testCaseData;
    }
}

In case something went wrong during the retrieval of the tests the infrastructure will be tried to TearDown to not leave the application on the cluster in a weird state that might hinder redeployment the next time the tests are run. The test runner interface that is implemented by the stateful service exposing the tests is fairly straight forward. It inherits from IService so that in can be exposed as a remoting service.

public interface ITestRunner : IService {
    Task<string[]> Tests();

    Task<Result> Run(string testName);
}

For the test names, no particular data contract is required since the test name can be represented as a string containing the fully qualified name. For the individual test runs a dedicated data contract is required that marshals the result of the tests back to the client. When an exception occurs inside a test run the exception needs to be marshaled to the client. With the data contract serializer, it is required to declare each type that might be exposed as an exception on the data contract. In the testing repo, I restricted the exception to AssertionException, IgnoreException, and InconclusiveException. When NUnit Asserts fail an AssertionException is raised. When a test is skipped an IgnoreException is raised. When a test is marked as inconclusive an InconclusiveException is raised. Since the client proxy test is run inside NUnit (in the console, on the build server or inside Resharper) these exceptions can simply be rethrown.

[DataContract]
[KnownType(typeof(AssertionException))]
[KnownType(typeof(IgnoreException))]
[KnownType(typeof(InconclusiveException))]
public class Result {
    public Result(string output = null, Exception exception = null) {
        HasOutput = !string.IsNullOrEmpty(output);
        Output = output;
        Exception = exception;
        HasException = exception != null;
    }

    [DataMember]
    public Exception Exception { get; set; }
    
    // other members omited
}

In the next post, I will cover the deployment step using conventions. Stay tuned and in the meantime: Be nice to your cluster.