TL;DR: Reading bytes from a queue is the easy part of a message pump. Things get difficult when message handling writes to a database, sends more messages, fails halfway through, moves to another broker, or runs in a cloud service with different transaction semantics. A small infrastructure project can quietly turn into a platform commitment.
None of this looked particularly scary when we started.
It looked like a loop: read from a queue, deserialize a message, find the handler, and call it. We added bounded concurrency when load arrived and graceful shutdown when deployments got rough.
Then people start using it, and the less obvious requirements show up.
Every message needs logging, failures need exception details, and successful processing needs auditing. Handlers need dependency injection scopes. Some operations must run in a transaction or produce more messages. Before long, the loop starts to look like middleware.
This is the point where I would slow down today. It is no longer just a helper; it defines how the system behaves when things fail.
The loop grew middleware
Business handlers rarely run alone. They need cross-cutting behavior around them. That behavior starts with logging and error handling, but it usually grows into auditing, tracing, metrics, retries, dependency injection scopes, and transaction boundaries.
The sample repository has a deliberately compact “how it probably looks like in reality” version. It creates a transaction and reads the message inside it. It then deserializes the payload, creates a child service provider, builds and runs the middleware, completes the transaction, and releases the concurrency slot.
The complete version with middleware is in ThePumpHowItProbablyLooksLikeInReality.cs.
async Task FetchAndHandleAndReleaseWithMiddleware(SemaphoreSlim semaphore, CancellationToken token = default) {
using (var transaction = CreateTransaction()) {
var (payload, headers) = await ReadFromQueue(token, transaction).ConfigureAwait(false);
var message = Deserialize(payload, headers);
using (var childServiceProvider = CreateChildServiceProvider()) {
try {
var middlewareFuncs = new Func<HandlerContext, Func<HandlerContext, CancellationToken, Task>, CancellationToken, Task>[] { Middleware1, Middleware2 };
var middleware = FlextensibleMiddleware(childServiceProvider, middlewareFuncs, token);
await middleware(message).ConfigureAwait(false);
transaction.Complete();
}
catch (Exception) {
// Just log?
}
finally {
semaphore.Release();
}
}
}
}
I left the comment in the catch block deliberately vague. “Just log?” sounds like an implementation detail, but it is a policy decision. Should the message be retried, moved to an error queue, acknowledged and skipped, or rolled back? How often, with which delay, and who gets alerted?
The pump has to answer those questions because it coordinates broker state with the side effects produced by application code.
The transaction boundary bites back
In the original system, Microsoft Message Queuing and SQL Server could participate in distributed transactions through the Distributed Transaction Coordinator. With a transaction scope and a two-phase commit protocol, the system could coordinate the receive, database work, and outgoing sends.
That gives a strong guarantee. Either the receive, database writes, and outgoing messages commit together, or they do not commit.
It also adds coordination cost. The transaction manager has to talk to the resources involved, and each resource has to agree on the outcome. That extra round trip and coordination can reduce throughput sharply.
This is often the first uncomfortable trade-off. Teams want the safety of a coordinated transaction and the speed of a simpler broker operation. The choice of broker decides how much of both they can have.
Receive-only changes the story
Some brokers do not coordinate the receive, database update, and outgoing sends into one transaction. RabbitMQ, Amazon Simple Queue Service, and Azure Storage Queues are common examples of systems where the receive side has its own acknowledgement model.
There is nothing inherently wrong with that model, but its guarantees are different.
Suppose a handler receives a message, writes to the database, and then fails before acknowledging the broker message. The broker redelivers the message later. The database write already happened. If the handler is not idempotent, the second attempt can create the same business effect twice.
The reverse can also happen. The handler sends a message to one destination, then fails before finishing the rest of the work. A downstream consumer can observe a message that represents work the original handler did not fully complete.
People often call those ghost messages. The name is informal, but the risk is real: a message escapes from a business operation that later rolls back or fails.
Cloud queues bring their own rules
Cloud brokers add another layer of semantics. Some queues use a visibility timeout. When the pump receives a message, the broker hides that message for a period. If the handler does not finish and delete the message before the timeout expires, the broker can make it visible again.
That means long-running handlers need renewal logic. The pump may have to extend the visibility timeout while business code is still running. Renew too aggressively and failures take longer to recover. Renew too little and duplicate processing appears while the first handler is still working.
Other brokers provide features that help with outgoing messages. Azure Service Bus, for example, has transaction support for sending messages through the incoming entity by using send-via behavior. That can reduce ghost-message risk for outgoing broker messages.
It does not make your database write magically part of the same broker operation. You still need idempotent business logic where your handler changes application state.
Cost sneaks into the loop
On-premises queues often make teams think in throughput, disk, memory, and operations work. Cloud queues add a direct cost model. Sends, receives, renewals, management calls, topic operations, and connections can all show up in billing or capacity limits depending on the provider and tier.
The exact prices change, so baking assumptions from a slide or a blog post into the pump is a bad idea. The underlying design pressure remains: every broker operation adds latency, and many of them also add cost.
Batching becomes a design concern. A loop that sends one outgoing message per broker call is easier to write, but it pays latency and operation cost for every send. A batching sender can group messages, but then it has to respect payload limits, destinations, transaction rules, and failure behavior.
Unfortunately, batching becomes transport-specific rather quickly. The rules for one broker rarely carry over unchanged to another.
Pub/sub adds topology
Sending commands to a known queue is the simpler case. Publish/subscribe adds topology.
Some brokers have topics and subscriptions as first-class concepts. Some have only queues and require another service, table, or convention to map a published event to subscribers. Once the pump supports publish/subscribe, it also needs routing rules, subscription storage, topology deployment, and a way to evolve those rules without losing messages.
At that point, the work extends well beyond code into documentation and operations. Someone has to know which event goes where, who owns the subscription, what happens during deployment, how to monitor dead-letter queues, and how to answer the uncomfortable question: can production handle the next customer?
Our little pump had become a platform.
A note on perspective: The events described in this post happened before I joined Particular Software. Today, I work on NServiceBus, so I have certainly developed opinions about the operational costs of owning messaging infrastructure. Building a message pump myself was still one of the most educational experiences of my career. I do not regret doing it. It simply gave me a better appreciation for the trade-offs involved when deciding whether to build or adopt an existing solution.
Build or use what works?
Writing a message pump taught me more about distributed systems than most books I had read at the time. I would still recommend building one as an exercise, even if you never intend to run it in production. Build a small one against your favorite broker. Make it read, deserialize, dispatch, retry, limit concurrency, shut down cleanly, and publish a follow-up message.
Then break it.
Kill the process during a handler. Drop the database connection after the write but before acknowledgement. Send a batch that exceeds the broker payload limit. Let a visibility timeout expire. Add a second subscriber. Watch what happens.
Eventually I had to ask whether messaging infrastructure differentiated the business. Most teams benefit from using proven solutions for common infrastructure problems and putting their effort elsewhere. This is the argument behind Use What Works, and message pumps are a good example. The first loop is easy. The years of edge cases, operational behavior, documentation, and support are what cost you.
Framed that way, the make-or-buy decision becomes more honest. If your business needs unusual semantics and your team can own the complexity for years, building may be reasonable. If you need retries, routing, topology, observability, auditing, error queues, transport quirks, and documentation while the team is also expected to ship domain features, off-the-shelf infrastructure starts to look less expensive.
The first loop is easy. Owning every failure mode behind it is a different commitment.
Further reading:
Thank you Daniel for sharing the experience and the difficulties you came across. We used MSMQ in the first big project instead and made good experiences with it. Years later I was reading the book “Enterprise Integration Patterns” from Gregor Hohpe and Bobby Woolf. At that point in time many vendors offered their messaging solution (RabbitMQ, ZeroMQ, Kafka, NServiceBus, etc.) and I was never thinking of making it ourselves.
When you buy a solution, you have the challenge of choosing one and the vendor lock-in which every Architect is afraid of. And then came Dapr (https://dapr.io/) which offered the decoupling from a specific vendor and offering a simple interface to messaging.
MSMQ is still a workhorse and extremely reliable. Of course, it is tricky to get it running on modern cross-platform .NET and has too much OS lock-in. Would love to hear your experience with Dapr. I heard very mixed things.