It’s time for bi-temporal event sourcing in part eight of my event sourcing series. In this post, I’ll stick to the basics – at least I hope so. 😅 Bi-temporal event sourcing means that we have two timestamps associated with an event: The first timestamp tells us when the event entered the system, or when the system acquired the knowledge. We call this the application timestamp. The second timestamp states when the event takes effect – the effective timestamp.
This is great, but complicated. Let me show you…
Why do we need application and effective timestamps?
Up to now in this series, we have always stored events with only an application timestamp. That timestamp tells us when an event entered the system, when the system became aware of a fact. But we had a big limitation so far. Events get appended to the event stream at the current time.
When I want to add an event or fact that something happened in the past or will happen in the future, I need a second timestamp – the effective timestamp.
For example, we store organisation charts in our time-tracking application. Typically, when there is a reorganisation of the company, this information is known in advance. So we would like to have an event that states the new organisation chart and when it takes effect: “Next year, we’ll have a new department with three new teams.”
Furthermore, we need to be able to hire new people for positions in these new teams. When creating a new person in the system, I should be able to assign them to one of the new teams, provided their first day of work is after the new chart is in effect. So, we need a way to see the data at a specific point in time.
Timelines
We have a lot of such scenarios in our system, data that changes over time, and we need the ability to see the data at a specific point in time, either the past, present or future. That’s why we introduced the concept of timelines in our system. Given any stream of bi-temporal events (events with application and effective timestamps), we have a common implementation to create a timeline.
Timelines are great for things that come into existence, change over time, and may die (or better be deleted). For example, an organisation chart has a defined start – when the company was founded or the date of a reorganisation. And it probably will change a couple of times (new teams or reorganisations). And it might be deleted when it’s replaced by another organisation chart completely (or the company dies – but then nobody cares about time-tracking anyway).
When we need the data at a specific point in time, we can ask the timeline. It will search for the correct phase on the timeline and return its data. Note that the timeline might not return a value when we ask for a time before the data was created or after it was deleted.
Different granularity
In most cases in our application, the application and effective timestamps have a different granularity. Application timestamps are always DateTimes (Date and time with as high a precision as our tech stack supports). Effective timestamps are often a date because data can only change on a given day. For example, an organisational change takes effect on the first of the next year.
Our timeline projection code works with any type for application and effective as long as it supports comparison. This makes testing a bit easier because we can test using integers.
Bi-temporal projection
To project a timeline from a bi-temporal event stream, we need to sort the events first. We order events by effective time, with application time used as a tiebreaker. That means we don’t use the natural ordering of events in the event stream (by application), but instead order them by their effective timestamp.
If there are multiple events with the same effective timestamp, we use the application timestamp as a secondary sort key. For events with the same effective timestamp, the most recently added event must be last, so the projected data reflects the latest event. This results in phases of zero duration. We decided to keep them because then the timeline still reflects the whole history of the data.
To sum up
Bi-temporal event sourcing is needed when we want to enter data that takes effect in the past, present, or future – not just when it enters the system. It also allows us to query data at a specified point in time. This is achieved by introducing a second timestamp (effective) in addition to the timestamp when the event was appended (application).
That’s the conceptual part for today. In the following blog posts, I’ll explore some challenges regarding bi-temporal event sourcing and how we solved them.
Deep Dive
Let’s take a look at the organisation chart/organisation form code in our system. As always, I don’t explain the details, but I hope you get an impression. More details will follow in later posts.
When I want to project the events of an organisation form, I need to call our TimelineProjector with a configuration:
The configuration specifies how events should be projected. First, we specify that we want to use the SortAndProject algorithm (just accept this for now, stuff for a later post). Then we specify the function that tells the timeline projector how events should be interpreted (CreateUpdateDelete: create a timeline using the definitions given by getProjectionAction) and how to obtain the application timestamp for an event (here, use the Application field).
The getProjectionAction function specifies how the timeline projector should treat each possible event in the event stream.
An OrganisationFormCreated event maps to a CreateProjectionAction.Creates. So after that event’s effective, data is available.
OrganisationUnitAdded, OrganisationUnitShifted, and OrganisationUnitRemoved tell the projector to update the current data and to create a new phase with updated data. Updates takes the effective timestamp and a function of the form previous -> updated as arguments (that’s hidden in the example because I didn’t show the functions changing the form).
The ModifyPerpetually events are a story for a later blog post (yeah, cliff hanger – everybody loves them!).
I also left out the actual code modifying the organisation form. “Just” some recursive tree manipulation stuff.
To make the picture a bit more complete, here is the definition of the OrganisationFormEvent. I didn’t include all the details of all the union cases (because boring). The only important aspect is that all events that create, update, or delete must have an effective timestamp. And as you can see in the OrganisationFormRemoved case, we use a Workday as the granularity for the effective time axis. A workday represents a date but may differ from a calendar date because people can work past midnight.
I know, I know, you all want to see the implementation of the timeline projector. But, believe me, it’s too early on your bi-temporal event sourcing journey; you probably wouldn’t understand why it is the way it is. 😊 And this post would be far too long. And if you still insist, you can find an old version in an old post on this blog.
To access the organisation form at a specific point in time, we can use the Timeline module. Here an example of a command that loads a organisation form for validation:
Load the events for the organisation form with the given ID, project the events into a timeline, and get the data at the Effective point in time. If there is no form on that date, return a validation error.
