Pimping Unquote

Unquote is a great library for writing test assertions in F#. In our acceptance tests, we frequently check data structures for equality. When these data structures get bigger, reading test failure messages gets harder. That’s why I implemented an F# data structure differ around Unquote that lets me quickly see the differences.

The following example shows a simple test sing Unquote and its output:

[<Fact>]
let ``simple example`` () =
    let a = [ 1 ; 2; 3 ]
    let b = [ 1 ; 3; 4 ]

    test <@  a = b @>
[1; 2; 3] = [1; 3; 4]
 false

Alternatively, a simple comparison can be written as:

[<Fact>]
let ``simple example`` () =
    let a = [ 1 ; 2; 3 ]
    let b = [ 1 ; 3; 4 ]

    a =! b

The real power of Unqote shows when using quoted expressions:

[<Fact>]
let ``expression example`` () =
    let a = [ 1 ; 2; 3 ; 4 ; 5]
    let b = [ 2 ; 4 ; 6 ]

    test <@ (a |> List.filter (fun x -> x % 2 = 0) = b) @>

The error message contains the individual steps to get to the result:

a |> List.filter (fun x -> x % 2 = 0) = b
 [1; 2; 3; 4; 5] |> List.filter (fun x -> x % 2 = 0) = [2; 4; 6]
 [2; 4] = [2; 4; 6]
 false

This helps us a lot when we do some transformations on the data that we want to check because we can see the original data before and during the transformations. For example, when we get a Result from a function and we want to do a Result.map on the data before comparing it. If the test fails, we see the original data.

The problem with bigger data structures

Some of our tests work on rather big data structures like trees or nested lists of records containing lists. With such data structures, it quickly gets tricky to spot what actually is different.

Look at this example:

type Leaf = { Name : string }
and Node = { Name : string ; Children : Tree list }
and Tree =
    | Leaf of Leaf
    | Node of Node

[<Fact>]
let ``tree example`` () =
    let a =
        Node {
            Name = "a"
            Children =
                [
                    Leaf { Name = "leaf1" }
                    Node { Name = "b"
                           Children =
                               [
                                   Leaf { Name = "leaf2" }
                               ]
                         }
                    Leaf { Name = "leaf3" }
                ]
        }
    let b =
        Node {
            Name = "a"
            Children =
                [
                    Leaf { Name = "leaf1" }
                    Node { Name = "c"
                           Children =
                               [
                                   Leaf { Name = "leaf2" }
                               ]
                         }
                    Leaf { Name = "leaf3" }
                ]
        }

    a =! b

The result of running the above test:

Node
   { Name = "a"
     Children =
               [Leaf { Name = "leaf1" };
                Node { Name = "b"
                       Children = [Leaf { Name = "leaf2" }] };
                Leaf { Name = "leaf3" }] } = Node
   { Name = "a"
     Children =
               [Leaf { Name = "leaf1" };
                Node { Name = "c"
                       Children = [Leaf { Name = "leaf2" }] };
                Leaf { Name = "leaf3" }] }
 false

I kept the sample short for the sake of a blog post, but I hope you can see why it gets difficult to see subtle differences quickly.

Let’s pimp it!

Let’s change the test to this (only a = is added so that we use the ==! operator instead of =!):

[<Fact>]
let ``pimped tree example`` () =
    let a =
        Node {
            Name = "a"
            Children =
                [
                    Leaf { Name = "leaf1" }
                    Node { Name = "b"
                           Children =
                               [
                                   Leaf { Name = "leaf2" }
                               ]
                         }
                    Leaf { Name = "leaf3" }
                ]
        }
    let b =
        Node {
            Name = "a"
            Children =
                [
                    Leaf { Name = "leaf1" }
                    Node { Name = "c"
                           Children =
                               [
                                   Leaf { Name = "leaf2" }
                               ]
                         }
                    Leaf { Name = "leaf3" }
                ]
        }

    a ==! b

When we run this test, the output looks as follows:

Delta:
 Node/Children[1]Node/Name b = c

Unquote Message:
 Node
   { Name = "a"
     Children =
               [Leaf { Name = "leaf1" };
                Node { Name = "b"
                       Children = [Leaf { Name = "leaf2" }] };
                Leaf { Name = "leaf3" }] } = Node
   { Name = "a"
     Children =
               [Leaf { Name = "leaf1" };
                Node { Name = "c"
                       Children = [Leaf { Name = "leaf2" }] };
                Leaf { Name = "leaf3" }] }
 false

The difference is easy to spot because the differences are listed directly at the top of the error message – followed by the Unquote error message. Woohoo!

Implementing the differ

First, we need a custom operator:

[<AutoOpen>]
module UnquoteOperators

let inline (==!) a b = UnquoteExtensions.diff a b

The custom operator simply delegates to the diff function:

module UnquoteExtensions

open Microsoft.FSharp.Reflection
open Xunit.Sdk
open Swensen.Unquote
open FSharpx

type ListIterator =
    static member GetValues<'a>(value : obj) =
        let casted : 'a list = downcast value
        casted |> List.toArray |> Array.map (fun v -> v :> obj)

let rec getDifferences (a : obj) (b : obj) breadcrumb =
    let t = a.GetType()
    if a = b then
        ""
    elif (t.IsGenericType && t.GetGenericTypeDefinition() = typedefof<FSharp.Collections.list<_>>) then
        let method = typeof<ListIterator>.GetMethod("GetValues").MakeGenericMethod(t.GetGenericArguments() |> Array.exactlyOne)
        let a_values : obj[] = downcast method.Invoke(null, [| a |])
        let b_values : obj[] = downcast method.Invoke(null, [| b |])

        if a_values.Length <> b_values.Length then
            let toString = Array.map (fun v -> $"{v}") >> String.concat ";"
            $"{breadcrumb} [{a_values |> toString }] = {b_values |> toString}"
        else
            Array.zip a_values b_values
            |> Array.mapi (fun i (x,y) ->
                if (x <> y) then
                    getDifferences x y (breadcrumb + $"[{i}]")
                else
                    "")
            |> Array.filter (fun x -> x.Length > 0)
            |> String.concat "\n"

    elif (FSharpType.IsUnion t) then
        if (FSharpType.GetUnionCases(t).Length = 1) then
            $"{breadcrumb} {a} = {b}"
        else
            let a_case = FSharpValue.GetUnionFields(a, t)
            let b_case = FSharpValue.GetUnionFields(b, t)
            if (a_case = b_case) then
                ""
            else
                let aa = a_case |> fst
                let bb = b_case |> fst

                if (aa = bb) then
                    Array.zip (snd a_case) (snd b_case)
                    |> Array.map (fun (x,y) ->
                        getDifferences x y $"{breadcrumb}{aa.Name}")
                    |> Array.filter (fun x -> x.Length > 0)
                    |> String.concat ", "
                else
                    $"{breadcrumb} {aa} = {bb}"

    elif t = typeof<string> then
        $"{breadcrumb} {a} = {b}"

    else
        let properties = t.GetProperties()

        if (properties.Length = 0) then
            if a <> b then $"{breadcrumb} {a} = {b}" else ""
        else

            let allValues =
                    properties
                    |> Array.map (fun p -> p.Name, p.GetValue(a), p.GetValue(b))

            let mismatches =
                allValues
                |> Array.filter (fun (_, pa,pb) -> not(pa = pb))

            mismatches
            |> Array.map (fun (name, pa,pb) -> getDifferences pa pb (breadcrumb + $"/{name}"))
            |> String.concat "\n"

let diff a b =
    try
        a =! b
    with
    | :? TrueException as e ->
        let message = getDifferences a b ""
        raise (TrueException("Delta:\n" + message + "\n-----------------------------------------------\nUnquote Message:\n" + e.Message, false))

The diff function first uses normal Unquote (line 80). When Unquote throws an exception, we catch it and execute getDifferences (line 83).

getDifferences does some reflection magic to walk through the value and compare field per field, list item per list item and so on.

Please note that the implementation above is not complete – it currently just covers what we need in our code. But I hope you get the idea to extend it for your code if needed.

The breadcrumb is used to keep track of where we are in the structure and to provide an error message that points to the difference.

That’s all for this time – happy coding!

About the author

Urs Enzler

Add comment

By Urs Enzler

Recent Posts