Machine.Specifications – The Heisenbug

In my last post, I talked about me moving away from maintaining Machine.Specifications. Today I want to give you more insights into the decision process and what else I tried to save the project. When I was working for bbv Software Services AG, I was heavily using MSpec in multiple projects. I and also others in the company liked the lightweight syntax and the code-centric approach to specifications by example. We used it so much that it became together with SpecFlow one of our strategic open source library for specifications. We used SpecFlow in projects where the product owner or business analysts took the time to learn and write Gherkin. We used MSpec when the development team was writing the specifications (of course as closely as possible with the business).

MSpec was always a bit weak when it came to providing different examples for the same spec. Traditional unit testing frameworks calls these row tests, i.ex. in xunit they are called Theory and in NUnit TestCase. Urs Enzler had a nice idea and submitted it as a feature request. Fabian Trottmann and I started working on that. Fabian started fresh from the university at bbv Software Services AG and helping me out on MSpec was his first task. He did an amazing job. At the same time, I began decoupling the MSpec core into multiple releasable components into separate repositories. Until today, we haven’t merged Fabian’s changes because we feared the number of unintended bugs we’d introduce because it was so darn hard to test the plugin for Resharper. Then there came the Heisenbug, which made me almost lose my sanity.

On the way to decouple runner, reporting and all other components of the core, I wrote a dynamic abstraction that marshals calls and information back and forth between the runner the app domains created in the core. When I debugged the Resharper plugin, all MSpec tests were executed properly, and the state reported back to Resharper was correct. As soon as I used the release version of the same plugin it didn’t work. I invested countless hours and late nights into that. And every time I changed the plugin I had to restart Visual Studio in a separate hive and reattach the debugger. Thanks to Igal, I found OzCode and it’s cool quick attach feature that saved me a bunch of time, but it was still painful. That whole process drained me out. The desire to really nail this, staying up late after a full workday and not being able to pull it off made my spider senses going off. I self-inflicted a few week break from MSpec because I felt I couldn’t continue like that. In the end, it was this tiny little component that was missing

    [SolutionComponent]
    public class RecursiveMSpecTaskRunnerRegisterer
    {
        public RecursiveMSpecTaskRunnerRegisterer(UnitTestingAssemblyLoader loader)
        {
            loader.RegisterAssembly(typeof(RecursiveMSpecTaskRunner).Assembly);
        }
    }

That assembly loading made Resharper correctly deserialize the tasks the MSpec runner was creating. And voilà it worked, and I felt like I’ve been shaving a herd of Yak recently! I started looking for alternative ideas how to overcome the issues I was facing with the plugin and how to progress faster with MSpec without losing the terse syntax it provided for users. Stay tuned for more updates in future posts!