TL;DR: A queue can remove temporal coupling between a front end and a back end, but something still has to read the messages and run the business code behind them. That something is the message pump. Processing one message at a time protects downstream systems but increases queue wait time. Fire-and-forget processing looks faster until concurrency runs away. In practice, the pump needs an explicit limit.
The medical invoicing system already worked. That made the request harder, not easier.
It handled medical invoicing for doctors, ran across a collection of servers, talked to SQL Server, and carried years of production decisions. You could call it a legacy mess if you wanted. The business called it a system that paid the bills. And they wanted it to keep doing exactly that.
Then the business wanted to move from medical invoicing into general-purpose invoicing for web shops. Same core idea, they said. Just invoices. Also, there was not much money for a full rewrite.
That changed the failure model. Doctors had working hours. Batch processing could happen at night. Web shops do not care about that schedule. A customer can click buy at any time, and the shop wants to know what happened.
The old architecture had too much temporal coupling. The front end, back end, and database all had to be available at the same time for the operation to succeed. If the database was unavailable, the failure flowed back up the stack. That was already inconvenient for the medical system. For web shops, it would become a business problem.
Start with one refund
We did not try to rewrite the whole system. That matters. A large legacy system with years of behavior does not become safe because a team names the next version after a new architecture style.
Instead, we picked one use case: invoice refunds. It was small enough to isolate, useful enough to offer to web shops, and connected enough to teach us how the old system would behave when messages entered the picture.
The new flow captured the refund as a command and put that command into a queue. The back end consumed the command when it could, performed the database work, and published an event when the refund state changed. A subscriber updated a query cache near the front end so polling web-shop clients could ask for the current state without depending directly on the back end.
The queue buffered load, but more importantly, it captured intent. The command represented work the business wanted done, even if the database was slow, the back end was being deployed, or a transient timeout happened in the middle.
The first queue looked innocent
The first receiving loop looks almost too simple: read one message from the queue, deserialize it, handle it, and repeat until the process is asked to stop.
The complete sequential sample is in ThePump.cs.
public void Start() {
tokenSource = new CancellationTokenSource();
var token = tokenSource.Token;
pumpTask = ((Func<Task>)(async () => {
while (!token.IsCancellationRequested)
{
var (payload, headers) = await ReadFromQueue(token).ConfigureAwait(false);
var message = Deserialize(payload, headers);
await HandleMessage(message, token).ConfigureAwait(false);
}
}))();
}
This version has a good property: it is easy to reason about. The handler completes before the pump reads the next message. If the handler fails, the pump has a clear place to decide whether to roll back, retry, or move the message somewhere else.
But that safety has a cost. If each handler takes one second and there are ten messages in the queue, the tenth message waits behind the first nine. That waiting time is not visible in the handler code, but users still feel it.
In messaging systems, that delay is part of critical time: the time from sending a message until the system finishes processing it. Critical time includes the network send time, the queue wait time, and the processing time. The first pump keeps processing simple but lets queue wait time grow under load.
Then rush hour happened
The tempting fix is to stop awaiting the handler work inside the loop. Start the work, go back to the queue, and read the next message.
The fire-and-forget version is in TheConcurrencyPump.cs.
pumpTask = Task.Run(() => {
while (!token.IsCancellationRequested)
{
FireAndForget(FetchAndHandleMessage(token));
}
});
That small change removes the sequential wait along with the only limit on concurrency.
What surprised us most was that none of these problems appeared during development. They only showed up once success arrived and the queues actually filled up.
If the queue has 20,000 messages and the broker can deliver them quickly, this pump tries to start 20,000 handlers. The code may look asynchronous, but the downstream systems are not infinite. A database has connection pools. A third-party application programming interface has rate limits. A process has memory, threads, sockets, and garbage collection pressure.
The queue was supposed to hold excess work outside the process. Unbounded concurrency pulls that excess work back into the process and then pushes the pain into the weakest downstream dependency.
Technically, the pump has more throughput. In practice, it has merely moved the bottleneck somewhere more painful.
The semaphore became the toll booth
A better pump needs an explicit concurrency boundary. In .NET, SemaphoreSlim is a practical fit because it supports asynchronous waits. The pump waits for a slot before reading and handling another message. When handling finishes, the slot is released.
The bounded concurrency version is in TheLimitingConcurrencyPump.cs.
semaphore = new SemaphoreSlim(MaxConcurrency);
pumpTask = ((Func<Task>)(async () => {
while (!token.IsCancellationRequested) {
await semaphore.WaitAsync(token).ConfigureAwait(false);
FireAndForget(FetchAndHandleAndRelease(semaphore));
}
}))();
The release belongs in a finally block. If the handler throws and the slot is not released, the pump eventually stops reading messages forever.
async Task FetchAndHandleAndRelease(SemaphoreSlim semaphore, CancellationToken token = default) {
try {
var (payload, headers) = await ReadFromQueue().ConfigureAwait(false);
var message = Deserialize(payload, headers);
await HandleMessage(message).ConfigureAwait(false);
}
finally {
semaphore.Release();
}
}
With that limit in place, the pump starts behaving like infrastructure. Its contract with the rest of the system is explicit: it can process several messages at once, up to a known limit.
That limit should come from the system you need to protect. If the database connection pool has 100 usable connections, the message pump should not pretend it can safely run 1,000 database-heavy handlers at once. If a downstream service allows 200 concurrent calls, the pump needs to respect that too.
Shutdown got less simple
Concurrency also changes shutdown. A sequential pump only needs to stop reading and wait for the current handler. A concurrent pump may have several handlers already running.
The sample waits until every semaphore slot has been returned before disposal:
while (semaphore.CurrentCount != MaxConcurrency) {
await Task.Delay(50).ConfigureAwait(false);
}
That detail is easy to overlook. Without a graceful shutdown path, the process can be killed while a handler is halfway through a database update, halfway through a send, or halfway through both. The broker might redeliver the message later, but the side effects that already happened do not disappear.
A message pump has to think about the work it has accepted, not only the messages still waiting in the queue.
What should you take away?
The queue solved a real architectural problem. It let the front end accept business intent without being temporally coupled to the back end and database. That bought us deployment room, failure isolation, and a way to absorb uneven traffic.
But the pump became part of the architecture too. The sequential version made critical time grow. The fire-and-forget version moved too much work into memory. The bounded version made the trade-off explicit.
Messaging does not remove limits. It lets you decide where to put them.
Once the pump had a concurrency limit, it felt production-ready. Then transactions entered the room. More about that in the next post.
Further reading:
