Event sourcing is easy. Just store what happened to a thing as events, and when you need the thing’s state, project the events. Unless it is not because there are misplaced or duplicated events in the event stream. In the tenth part of my event sourcing series, we’ll look at why events can be misplaced and how to handle them.
What a misplaced event is
I guess I invented the term misplaced event while writing this blog post. I didn’t find an existing term for this.
A misplaced event is an event that is at the wrong place on the event stream. Over our eleven years of using event sourcing, we encountered a couple of these.
How events get misplaced or duplicated
In our system, there exist two reasons for events to get misplaced.
The first reason is that a user enters the wrong effective date for a bi-temporal event. Of course, we try to detect these during command validation, but there are scenarios when this is really hard – especially when the user can undo previous actions. For example, in our time-tracking system, users can define an organisational chart. A so-called organisation form that consists of multiple organisation units (as explained in the last two posts in this series). A unit has a creation date from which on it exists in reality. But maybe this date changes or was entered incorrectly. So we need to be able to change this date. Furthermore, there might be update events that change a unit’s property. Now, it is possible that, after a couple of changes, the update event is before the creation date.
The more common reason for misplaced or duplicated events, however, is a technical one: our system misbehaved. Maybe the backend was unresponsive, we programmed a bug into the code, there was a network hiccup that led to retries by the client and/or the user. Of course, we try to provide a stable, responsive, quality system. But sometimes things go wrong.
How to deal with misplaced or duplicated events
We try to detect misplaced events and ignore them. Maybe you remember that our projector does lifetime management. The projector tracks the creation and deletion of the thing. The projector distinguishes among creation, update, and deletion events. So, the projector can ignore events in a couple of situations:
- Update events before any creation event: the update event gets ignored because there is no current value to update.
- Update events after deletion event: gets ignored because there is no need to update the deleted data.
- Deletion events before any creation event are ignored.
- Duplicated creation events, or multiple creation events without a deletion event in between: only use the first creation event.
- Duplicate deletion events without any creation events in between get ignored.
We can’t detect duplicate update events, but they’re typically not a problem because they just set the data to the same value again.
Another approach to preventing duplicate events would be to compare them. If two adjacent events are equal (except for the event ID), we could ignore one of them. However, there are rare scenarios in which these equal events are both real. While deduplicating these events into a single event does not change the resulting value, it could matter when we are interested in the number of events for some reason.
Looking back, we had a good experience with our approach. And keep in mind that misplaced or duplicated events should be rare cases.
Summary
When an event-sourced system runs for many years, sometimes things go wrong. One consequence we saw in our system is that there are events that are misplaced or duplicated. Even if rare, we need a way to handle these events. We decided to use our projector’s lifetime management capabilitiesto handle these cases.
Deep dive
Let’s look at some tests again:
These tests check that updates that are before a create event or after a deletion event are ignored.
I know, you’ve already seen such tests, so let’s expand the deep dive to the parsing of the events and the projection:
I simplified the code a bit by removing things we haven’t discussed yet (the red dots indicate the missing lines).
The parsing is done by simply splitting, trimming, and matching the input to fixed constants. Simple and good enough.
The EventsForProjection type is used to give a list of events more meaning.
A value of the type EventsForProjection holds a list of events from a single event stream that can be projected into a single value (or a single timeline). A value of the type EventsForProjections holds events from several event streams and must be projected into several values (or timelines). To separate the individual streams, we need to pass a function to the projector that specifies how to identify events belonging to the samestream. Typically, this is a function that returns the event stream ID for an event:
When projecting multiple organisational forms, the projector first groups the events by their OrganisationFormId and then projects all the events per group.
The EventsForProjection.shake function in the parsing function changes the order of the events so that we know that the events are sorted correctly:
This way, we can write the events in the test data in the correct order, making the test easier to understand while still ensuring the implementation sorts correctly.
