Our journey to F#: JSON serialization with a mix of C# and F#

There are many libraries for JSON serialization in the .Net realm. The best known are probably Newtonsoft’s Json.NET and System.Text.Json. Both can’t handle F# discriminated unions very well. There are also a couple of F# JSON libraries available like Thoth.Json or FSharp.Json. They are all great libraries, and choosing one is about making trade-offs.

However, our case is a bit complicated because we have a mix of C# and F# code. This results in object/value graphs that consist of a mix of basic types, C# classes, C# records, F# records, and F# discriminated unions. None of the above libraries – as far as we found – can handle a mix well. So we had to find our way to deal with it. And this blog post is about our solution.

Context

When we started our product TimeRocket, there was only .Net Framework and the go-to JSON library was Newtonsoft’s Json.NET. So that is what we used. When we switch from .Net Framework to .Net Core, there was another option, System.Text.Json. However, all our classes used for communication between the backend and the client are immutable, with a single constructor. No problem for Json.NET, not supported by System.Text.Json back then. Nowadays it is, but you have to use attributes on your classes. AND I HATE THAT! A class should not have to know whether it is ever JSON serialized or not, never. Anyway, we had a running system and didn’t want to change everything over to System.Text.Json.

Then, we started to add F# code. One of the reasons were discriminated unions. They are great for modelling a domain. They are not so great regarding JSON serialization – when you want to get them into TypeScript code on the client-side. I wrote about our approach to get type-safety across .Net and TypeScript here. So we got that working. But one problem remained, how to serialize a mix of C# and F# classes with records and unions?

Mixing Json.NET with Thoth.Json

We are very happy with Json.NET on the C# side and with Thoth.Json on the F# side. Both libraries are rather simple in use and allow us enough possibilities to tweak their behaviour. So we decided to mix them together.

A Json.NET converter in F# that calls Thoth

The following code shows our Json.Net converter (written in F#) that reacts when a type should be (de)serialized that comes from one of our F# assemblies:

module JsonConverters

open System
open System.Reflection
open Calitime.Employees
open Newtonsoft.Json
open Newtonsoft.Json.Linq
open Thoth.TypeScript.Json.Net

// --> add new F# assemblies here <--
let isFSharpAssembly (assembly : Assembly) =
    let assemblyName = assembly.GetName().Name

    assemblyName = "Calitime.TimeRocket.Core.ActivityTime"
    || assemblyName = "Calitime.TimeRocket.Core.Expenses"
    || assemblyName = "Calitime.TimeRocket.Fundamentals"

let extra (settings : JsonSerializerSettings) =
    Extra.empty
    |> Extra.withCustom
         (fun (employeeRepresentation : EmployeeRepresentation) ->
                JsonConvert.SerializeObject(employeeRepresentation, settings) |> JToken.Parse)
         (fun (_ : string) _ -> Error ("cannot deserialize EmployeeRepresentation inside F# record", FailMessage("cannot deserialize EmployeeRepresentation inside F# record")))


type Encoder() =
    static member Encode<'T>(v : 'T, settings : JsonSerializerSettings) =
        JsonConversion.Web.encodeWithExtra (extra settings) v

let encoderMethod = typeof<Encoder>.GetMethod("Encode")

type Converter() =
    inherit JsonConverter()

    /// if the type is from one of our F# assemblies then this converter should convert it
    override self.CanConvert(objectType : Type) =
        isFSharpAssembly objectType.Assembly


    override self.WriteJson(writer : JsonWriter, value : obj, serializer : JsonSerializer) =
        let settings = JsonSerializerSettings()
        settings.Formatting <- serializer.Formatting
        settings.DateFormatHandling <- serializer.DateFormatHandling
        settings.DateParseHandling <- serializer.DateParseHandling
        settings.DateTimeZoneHandling <- serializer.DateTimeZoneHandling
        settings.ContractResolver <- serializer.ContractResolver
        settings.Converters <- serializer.Converters

        let makeGenericMethod = encoderMethod.MakeGenericMethod(value.GetType())
        let json = downcast makeGenericMethod.Invoke(null, [| value ; settings |])
        json |> writer.WriteRawValue


    override self.ReadJson
        (reader : JsonReader, objectType : Type, _existingValue : obj, _serializer : JsonSerializer)
        =
        let json =
            match reader.TokenType with
            | JsonToken.String -> $"\"{downcast reader.Value}\""
            | _ -> JObject.Load(reader).ToString()

        let value = JsonConversion.Web.decode objectType json

        match value with
        | Ok v -> v
        | Error error -> failwith error


let getAllJsonConverters () = [| Converter() |]

Line 10: We use the assembly name to check whether the specified is an F# assembly und its types should be handled by this converter. if you know a better – and performant – approach, please let me know in the comments.

Line 18: Thoth can be configured/extended with so called extras. An extra specifies how a type is serialized and deserialized. In the code above there is an extra for serializing EmployeeRepresentations. They only need to be serialized, never deserialized. An extra is needed for EmployeeRepresentation because it is a C# type that cannot be serialized by Thoth. So we delegate the serialization to Json.NET. We also pass along the settings (more about them later). We add an extra for every C# type that is below an F# type in an object/value-graph.

We have more extras, but they are defined in a different place (some are shown later). The reason this extra is here and not with the others is the dependencies between our assemblies.

Line 26: The Encoder is used to make the call to encodeWithExtra with the correct generic type. v needs to be of the correct compile-time type so that Thoth can serialize it correctly. The Json.NET settings are passed along to the extra function. The function JsonConversion.Web.encodeWithExtra is shown below.

Line 30: We use encoderMethod to get the Encode method of the Encoder. Note that this is only executed once.

Line 32: The Converter inherits from Json.NET’s JsonConverter so that we can add it to our Json.NET configuration.

Line 36: We can convert an object if the object’s type is from an F# assembly.

Line 40: When serializing a value, we need to propagate the JSON settings to Thoth. This is necessary when a F# type contains an C# type (like the EmployeeRepresentation shown above. The contained type needs to be serialized with Json.NET again. We create a new setting value and copy all relevant settings from the serializer. Then we create the generic encoder method for the type of the value to be serialized. Finally, we invoke the encoder method and get the JSON string back.

Line 54: When deserializing a value, we first check whether the current JSON token is a string or an object. If it is a string, we simply read the value. If it is an object, we have to read it into a JObject and get its string representation. This is not performant, but we need the JSON as a string so that we can pass it the Thoth. Note that this code is only run when we have an F# object inside an C# object. Therefore, we try to limit these cases in our code. Most object/value graphs are either purely C# or F#, and can be serialized and deserialized without these extra steps.

Then we pass the JSON string to Thoth for deserialization. If successful, we return the value, otherwise, we raise an exception. The JsonConversion.Web.decode function is shown below.

Line 69: The function getAllJsonConverters is used to setup our Json.NET settings and to add the above converter to the list of known converters.

Configuring Thoth.Json

The following code is our configuration wrapper around Thoth:

module JsonConversion

open System
open System.Globalization
open Calitime.Identifiers

module Database =
    open Thoth.Json.Net

    let writeLocalDate = NodaTime.Text.LocalDatePattern.Create("yyyy-MM-dd", CultureInfo.InvariantCulture).Format >> Encode.string
    let readLocalDate = Decode.string |> Decode.map(fun value -> NodaTime.Text.LocalDatePattern.Create("yyyy-MM-dd", CultureInfo.InvariantCulture).Parse(value).Value)
    let writeLocalTime = NodaTime.Text.LocalTimePattern.Create("HH:mm", CultureInfo.InvariantCulture).Format >> Encode.string
    let readLocalTime = Decode.string |> Decode.map(fun value -> NodaTime.Text.LocalTimePattern.Create("HH:mm", CultureInfo.InvariantCulture).Parse(value).Value)
    let writeGuid guid = guid.ToString().ToUpper() |> Encode.string
    let readGuid = Decode.string |> Decode.map(Guid.Parse)

    let private getExtra additionalExtra =
        additionalExtra
        |> Extra.withDecimal
        |> Extra.withCustom writeLocalDate readLocalDate
        |> Extra.withCustom writeLocalTime readLocalTime
        |> Extra.withCustom (fun guid -> guid.ToString().ToUpper() |> Encode.string) (Decode.string |> Decode.map(Guid.Parse))
        |> Extra.withCustom (fun (employeeId : EmployeeGuid) -> employeeId.Value |> writeGuid) (readGuid |> Decode.map(EmployeeGuid))

    let encode value =
        Encode.Auto.toString (0, value, caseStrategy = CaseStrategy.CamelCase, extra = getExtra Extra.empty)

    let encodePrettyPrint value =
        Encode.Auto.toString (4, value, caseStrategy = CaseStrategy.CamelCase, extra = getExtra Extra.empty)

    let encodeWithExtra (additionalExtra : ExtraCoders) value =
        Encode.Auto.toString (0, value, caseStrategy = CaseStrategy.CamelCase, extra = getExtra additionalExtra)

    let decode objectType json =
        Decode.Auto.LowLevel.fromString
            (json, objectType, caseStrategy = CaseStrategy.CamelCase, extra = getExtra Extra.empty)

    let decodeThrowing<'objectType> json =
        let result =
            Decode.Auto.LowLevel.fromString<'objectType>
                (json, typeof<'objectType>, caseStrategy = CaseStrategy.CamelCase, extra = getExtra Extra.empty)

        match result with
        | Ok value -> value
        | Error error -> failwith error

Note: we have two versions of Thoth that we use. One for (de)serialization for the communication with our TypeScript client (a pimped one) and the normal version for (de)serialization for persisting data in storage. See here for an explanation. Above, only the Database version is shown.

Line 10-15: These are functions for handling values that Thoth does not understand out of the box.

Line 17-23: Here we configure the basis extras for Thoth that are then used below.

Line 25: Serialize (encode) a value to JSON using the extras specified above.

Line 28: Human readable version of encoding.

Line 31: In case that additional extras are needed, this function can be used. The passed extras are added to the basic extras.

Line 34: Deserialize (decode) a value. This function returns a Result that reflects whether the deserialization was successful.

Line 38: Deserializes (decodes) a value and throws an exception if the deserialization was unsuccessful.

Note: Thoth also provides “manual” serialization and deserialization. The above code uses only the auto serialization features.

Conclusions

With the above code, we can serialize and deserialize object and value graphs that consist of both C# and F# types, including F# records and discriminated unions. Woohoo!

Find all blog posts about our journey to F# here.

This blog post is made possible with the support of Time Rocket, the product this journey is all about. Take a look (German only).