Machine.Specifications – The alternative xunit

In my last post, I showed how a Heisenbug can look like to you. In this post, I want to explore the different options I was considering to get rid of some of the pesky things around Machine.Specifications. During the work, I did for Machine.Specifications I asked myself more and more if it is really worth maintaining a whole ecosystem of tools when there are excellent frameworks available in the Open Source world. Why did I even look at alternatives? Why didn’t I just say, that’s it and move away from Machine.Specifications? To be honest, I felt responsible for the community and the framework. Because I profited from MSpec in the past I wanted to give something back. Not only to the people who previously maintained it but also to the community itself. On the other hand, I was curious how other testing frameworks were achieving and implementing things including the Resharper Runner.

Consider for example xunit. Brad Wilson and his community have been reimplementing the xunit framework quite a bit in the last few years. In fact, the structure became so modularized that you can completely swap out the test discovery, the test execution and in fact almost the whole test framework. Let’s look at some pseudo-code to describe my ideas for xunit. You as a user of the xunit version of MSpec would just need the following code in the AssemblyInfo.

[assembly: TestFramework("Machine.Specifications.Execution.MachineSpecificationsTestFramework", "Machine.Specifications.Execution.desktop")]

The Machine.Specifications.Execution.*.dll just needs to be placed into the AppDomain base directory, and then xunit automatically picks it up. The code that is executed by xunit looks nearly the following

    public class MachineSpecificationsTestFramework : TestFramework
    {
        public MachineSpecificationsTestFramework(IMessageSink diagnosticMessageSink)
            : base(diagnosticMessageSink)
        {
        }

        protected override ITestFrameworkDiscoverer CreateDiscoverer(IAssemblyInfo assemblyInfo)
        {
            return new MachineSpecificationsDiscoverer(assemblyInfo, this.SourceInformationProvider, this.DiagnosticMessageSink);
        }

        protected override ITestFrameworkExecutor CreateExecutor(AssemblyName assemblyName)
        {
            return new MachineSpecificationsExecutor(assemblyName, this.SourceInformationProvider, this.DiagnosticMessageSink);
        }
    }

    public class MachineSpecificationsExecutor : TestFrameworkExecutor<ITestCase>
    {
        public MachineSpecificationsExecutor(AssemblyName assemblyName, ISourceInformationProvider sourceInformationProvider, IMessageSink diagnosticMessageSink)
            : base(assemblyName, sourceInformationProvider, diagnosticMessageSink)
        {
        }

        protected override ITestFrameworkDiscoverer CreateDiscoverer()
        {
            return new MachineSpecificationsDiscoverer(this.AssemblyInfo, this.SourceInformationProvider, this.DiagnosticMessageSink);
        }

        protected override void RunTestCases(IEnumerable<ITestCase> testCases, IMessageSink executionMessageSink,
            ITestFrameworkExecutionOptions executionOptions)
        {
        }
    }

    public class MachineSpecificationsDiscoverer : TestFrameworkDiscoverer
    {
        public static readonly string DisplayName = String.Format(CultureInfo.InvariantCulture, "MachineSpecifications {0}", typeof(MachineSpecificationsDiscoverer).GetTypeInfo().Assembly.GetName().Version);

        readonly Dictionary<Type, IXunitTestCaseDiscoverer> discovererCache = new Dictionary<Type, IXunitTestCaseDiscoverer>();

        public MachineSpecificationsDiscoverer(IAssemblyInfo assemblyInfo, ISourceInformationProvider sourceProvider, IMessageSink diagnosticMessageSink, IXunitTestCollectionFactory collectionFactory = null)
            : base(assemblyInfo, sourceProvider, diagnosticMessageSink)
        {
		// ...
        }

        public IXunitTestCollectionFactory TestCollectionFactory { get; set; }

        protected override ITestClass CreateTestClass(ITypeInfo @class)
        {
            return new TestClass(TestCollectionFactory.Get(@class), @class); ;
        }

        protected virtual bool FindTestsForMethod(ITestMethod testMethod, bool includeSourceInformation, IMessageBus messageBus, ITestFrameworkDiscoveryOptions discoveryOptions)
        {
            var discoverer = new TestCaseDiscoverer(DiagnosticMessageSink);

            foreach (var testCase in discoverer.Discover(discoveryOptions, testMethod, null))
                if (!ReportDiscoveredTestCase(testCase, includeSourceInformation, messageBus))
                    return false;

            return true;
        }

        protected override bool FindTestsForType(ITestClass testClass, bool includeSourceInformation, IMessageBus messageBus, ITestFrameworkDiscoveryOptions discoveryOptions)
        {
            foreach (var method in testClass.Class.GetMethods(includePrivateMethods: true))
            {
                var testMethod = new TestMethod(testClass, method);
                if (!FindTestsForMethod(testMethod, includeSourceInformation, messageBus, discoveryOptions))
                    return false;
            }

            return true;
        }
    }

But the most important part is the test case discoverer. Responsible for finding test methods which can be executed by xunit.

    public class TestCaseDiscoverer : IXunitTestCaseDiscoverer
    {
        private readonly IMessageSink _messageBus;

        public TestCaseDiscoverer(IMessageSink messageBus)
        {
            _messageBus = messageBus;
        }

        public IEnumerable<IXunitTestCase> Discover(ITestFrameworkDiscoveryOptions discoveryOptions, ITestMethod testMethod,
            IAttributeInfo factAttribute)
        {
            if (testMethod.Method.Name.StartsWith("It_"))
            {
                yield return new XunitTestCase(_messageBus, discoveryOptions.MethodDisplayOrDefault(), testMethod);
            }
            yield break;
        }
    }

But here is also where the problems started. MSpec is built around delegates and statics. After the code is compiled all the reflection abstractions xunit has built-in cannot be used because the anonymous delegate methods get compiled to static methods and assigned to fields. The reflection abstractions of xunit detected those methods, but I couldn’t connect back properly to the fields and, therefore, lost the type information (is it an Establish, It or a Because?). Only relying on naming conventions would not be backwards compatible at all. Furthermore, I felt that because xunit has still a few fundamental assumptions built into it would be a good fit for the static context of MSpec and heavy customization would be required.

But on the other hand xunit had an active Resharper runner that is built by Matt Ellis. Because Matt works for Jetbrains he knows the inner workings of the SDK. That runner Matt is maintaining, usually also serves as THE early adopter when it comes to SDK changes. But let’s be honest. Also this runner is pretty much a one-man-show as far as I know. So if anytime Matt has enough or is no longer working for Jetbrains who would then maintain it?

In the next post of this series, I’m going to look into the next alternative – nunit.