Event Sourcing: Simple is often enough

This is the first post in a series about event sourcing. I’ll start with a very simple event sourcing implementation that is often good enough. Most of our event streams are implemented in this simple approach. In the following posts, the concepts will be extended to match additional requirements. I’ll touch on read models, consistency, long event streams, archiving, compensation, event skipping, lifetimes, and bi-temporal event sourcing.

Every post will explain the concepts and our (trade-off) decisions made, leading to our solution. So, I won’t present a general solution, but you’ll see how we tackled our problems and requirements, hopefully helping you with your decision-making.

In this post, we start with a simple event sourcing implementation.

Event Sourcing Introduction

Event sourcing means you store every change to an application’s state as a separate event. Each event describes what happened, not what the final state should be. The current state – the projection – is obtained by projecting all events in order.

Events are immutable, so once written, they are never changed*. Because events never disappear, you always have a complete history of everything that happened.

* There are exceptions we’ll look at in a later post in this series.

About the order of events

Typically, events are ordered by a so-called version or event number.

We use the timestamp when the event entered the system – the application timestamp. We made this decision because we weren’t sure how we would store the events (SQL Server, Document DB, …). We wanted a database-agnostic event number that is not based on a row ID or any other database-provided identifier.

The second reason is that we knew from the start that we would need to support bi-temporal event sourcing: events also have an effective date – the date when they take effect. In bi-temporal event sourcing, events also need to be sorted by their effective dates. So, we decided that it is easier for us to always sort by date, even in simple cases. The drawback is that sorting by dates is a bit slower than sorting by a number. Not much, however, because a date is also internally represented as a number.

Lifetime Information

Another unusual decision we made is that we track lifetime information as metadata. When projecting events, we annotate them as creation, update, or deletion. Moving this state tracking from the projections to the projection mechanism gives us a general approach to deal with deleted entries. Instead of every projection tracking its own state (created, deleted), the state is represented by a type: something like Existing<T>, Deleted<T>.

Again, this decision was primarily driven by our bi-temporal event-sourcing needs. But even for simple cases, it is nice to have.

We’ll come back to lifetime tracking in a later post.

Querying limitations

Because we need to project all events of an event stream to get the current state of a “thing”*, we have very limited query capabilities. Typically, we can only query an event stream by the ID of the “thing” it represents.

If we have other criteria for the lookup, we need to either add additional data to the events or introduce a read model. More on that in a later post.

• I don’t use the terms “entity” or “aggregate” because they carry too much meaning that’s not relevant to event sourcing.

But the performance!

With this simple approach, we always need to project all events in an event stream. This is, however, not a problem when the event streams are short.

In our experience, the bigger problem is typically query execution times – finding the right events, not the raw projection time. Of course, this depends on the projections done. We had one projection that manipulated a tree structure. This proved to be too slow.

Later in this series, we’ll look at strategies to keep event streams short, how to use read models and snapshot events, and the drawbacks of each. Whenever possible, I try to go with this simple approach.

Conclusions

This minimal approach to event sourcing is often sufficient for many use cases. In our system, most event streams are implemented this way.

However, it is not a good choice when:

• Data can’t be queried by ID
• Long event streams (the projections take too long)
• CPU-heavy projections (e.g. manipulating big tree structures)

To be continued in the next post…

An example in F# – for the ones who like a deep dive

This example shows how we implemented expense tracking in our system using event sourcing. This is our real production code.

I haven’t yet explained everything shown in the code samples. So don’t worry, I’ll cover them later in this series.

The events

Expenses are entered when an employee makes an entry.
Expenses then either get accepted or rejected.
Finally, accepted expenses are processed so the employee receives their money back.
And because humans make errors, expenses can be withdrawn, or the processing can be undone.

We define all the different event variants of an event stream in a discriminated union:

Regarding humans making mistakes: the ExpensesCreated type should be called ExpensesEntered. 🙈

An expense has a measure (currency, km by car, …), receipts, and some additional data.
We also keep track of who accepted or processed the expense, together with a notice.

Finally, the event record tells us who the expense is for, on what day it occurred, what kind of expense it was (expense position), and metadata about when the event entered the system (application) and what triggered it. The trigger is typically the ID of the executed command, linking this expense to the audit log.

The ExpenseId is the stream ID. Why the other IDs and the workday are here will become clear in the next post.

The Projection

Most samples I found on the internet perform the projection within the Aggregate; we only have a function. On one hand, it just feels more like functional programming; on the other hand, it allows us to easily apply different projections to the same events. Did I mention that composition in FP is easier than in OOP? 😅

The getProjectionAction function takes an event and returns a definition of what the projector should do. In the code above, we tell the projector that an ExpenseEntered event means that an expense was created, and we pass the projected expense along.

We continue by specifying what should happen for the other events:

In the case of an Accepted, Rejected or Processed event, we update the Status field of the projected expense. The function we have to define for an Updates gets the previous projection, and we return the updated one. I omitted the Withdrawn case because it’s very similar.

Note that the Status is not a simple flag. It holds the full history of when and by whom the expense was accepted, rejected and processed (PhaseList = a list of phases on a timeline with or without a value). A bit of event sourcing within event sourcing. This allows us to display the whole processing history to the user.

Undoing processed expenses is a bit more difficult:

We need to find the notice of the latest Accepted event so that we can append an Accepted with the same notice to the state.

Finally, we provide some functions to perform the projection:

Please note that the result of a projection is a Projection<T> that holds the metadata whether the requested data resulted in a thing that really exists (Exists<T>), was deleted (ByGone<T>), or never existed (NeverExisted<T>). Projection.current transforms a Projection<T> into an Option<T> that only has a value when the thing exists. Projection.allCurrent filters for existing expenses.

Don’t worry if you got lost in the details of lifetime tracking. I’ll return to this in a later post.

The apply function applies an event onto an already existing projection. Why we need this will be clear once we look at read models and how we use them.

That’s it for this post…