TL;DR: A visibility timeout is a lease. It tries to keep two consumers from working on the same queue message at the same time, but only for a limited period. If message handling takes longer than that period, the pump must renew the lease while business code continues to run. That sounds like a small addition until renewal needs scheduling, cancellation, shutdown behavior, mutable pop receipts, delete coordination, and a policy for what to do when renewal fails.
The first version of the message pump had one job: read a message and call a handler.
Then load arrived, so we bounded concurrency. Then failure arrived, so we had to think about transactions, acknowledgements, retries, and outgoing messages. By then the pump already looked less like a loop and more like infrastructure.
One more requirement tends to sneak in quietly: some handlers take longer than the queue’s visibility timeout.
That sounds easy to fix. Just renew the lease.
And then the pump grows again.
Start with the naive receive loop
Azure Queue Storage, like several cloud queues, does not remove a message when a consumer receives it. It hides the message for a configured visibility timeout. During that period, other consumers should not receive the same message.
The receiver then has a limited amount of time to finish the work and delete the message. If it does not, the message can become visible again and another receiver can process it.
This is a useful model. The queue does not have to trust the receiver forever. If the process crashes after receiving a message, the message eventually becomes visible again. Another process can pick it up and try again.
The visibility timeout is not just a retry delay. It is the queue’s lease on the message. While the lease is valid, the receiver has temporary ownership. When the lease expires, ownership can move to another receiver.
That lease protects recovery. It does not prove that processing finished.
Long-running handlers break the simple version
A short handler fits nicely inside the visibility timeout. Receive the message, run the handler, delete the message, and move on.
Now make the handler slower. It might call a partner application programming interface, wait for a database lock, or produce a large document. The work is usually fast, until one dependency stalls.
For example, an order message may generate an invoice PDF, call a slow invoicing backend, and store the resulting document. The handler is usually fast, but occasionally the backend stalls and the work outlives the visibility timeout.
If the handler runs for 12 seconds and the visibility timeout is 10 seconds, the first receiver is still working when the queue makes the message visible again.
This is not an Azure Queue Storage oddity. Any broker that uses lease-based delivery has some version of this problem. The broker needs a way to recover messages from crashed consumers. The timeout is that recovery mechanism.
The failure mode is uncomfortable because both receivers may be behaving correctly. Pump A is finishing the original work. Pump B sees a visible message and starts processing it. The queue has no way to know whether Pump A is slow, stuck, disconnected, or gone.
The pump needs to keep the lease alive while the handler runs.
Just renew the lease
The next version adds a renewal task. After receiving the message, the pump starts two concurrent activities. One runs the handler. The other periodically extends the visibility timeout.
The sample receiver in Receiver.cs uses a 10 second visibility timeout and schedules renewal halfway through that period. The excerpt below shows the delay and renewal call inside the renewal loop:
await Task.Delay(TimeSpan.FromSeconds(visibilityTimeout.TotalSeconds / 2), cancellationToken);
UpdateReceipt updateReceipt = await receiver.UpdateMessageAsync(
message.MessageId,
message.PopReceipt,
visibilityTimeout: visibilityTimeout,
cancellationToken: cancellationToken);
Renewing halfway through the lease is a common shape. It gives the pump some room for scheduler delay, network latency, and transient slowness before the current lease expires. The full loop also has to keep the updated message or receipt for the next renewal attempt.
This still feels manageable. The handler and the renewer are separate activities with different responsibilities. The handler completes business work. The renewer keeps temporary ownership alive. When the handler finishes, the pump deletes the message and stops renewing.
That is the happy path. Production code has to care about the paths around it.
Renewal cannot wait behind business work
A tempting implementation is to renew from inside the handler. For example, the handler could call a “renew lease” method before it starts a slow operation or between batches of work.
That puts infrastructure timing inside business code. It also assumes the handler will reach the renewal call on time. If the handler blocks on a database call, burns CPU, waits on an external service, or gets stuck behind its own retry policy, the renewal may not happen before the lease expires.
Lease renewal belongs to the pump because the pump owns the delivery contract. The handler should not need to know which broker delivered the message, how long the visibility timeout is, or which token proves ownership.
That separation has a cost. The pump now runs infrastructure work next to business work and must coordinate the two.
The pop receipt changes after renewal
Azure Queue Storage uses a pop receipt to prove that the receiver currently owns the message. The pop receipt returned by the receive operation is valid for that lease. When the pump renews the visibility timeout with UpdateMessageAsync, Azure Queue Storage returns a new pop receipt.
That detail matters. Deleting the message with an old pop receipt can fail because the old receipt no longer represents the current lease.
The sample code tracks that updated receipt outside the business handler. It associates the coordination token source for a received message with the latest UpdateReceipt:
private ConditionalWeakTable<CancellationTokenSource, UpdateReceipt> sourceToPopReceipt = new();
This is one implementation technique in the sample, not a recommendation for every pump. A dedicated in-flight message context that owns the current pop receipt, cancellation token, and renewal state would be just as reasonable in production code.
When renewal succeeds, the renewer stores the new receipt:
message = message.Update(updateReceipt);
sourceToPopReceipt.AddOrUpdate(coordinationTokenSource, updateReceipt);
Before deleting, the handler side checks whether a newer receipt exists and updates the message object it will use for deletion:
if (sourceToPopReceipt.TryGetValue(coordinationTokenSource, out var updateReceipt))
{
message = message.Update(updateReceipt);
}
await receiver.DeleteMessageAsync(
message.MessageId,
message.PopReceipt,
CancellationToken.None);
The ConditionalWeakTable is not the important part. The pump needs a safe place to store mutable lease state that changes while the handler is running. The receipt is broker state, not business state. It should not leak into the handler just because the pump needs it later.
Using a weak table also avoids making the receipt lifetime longer than the coordination object that represents the in-flight message. The exact storage mechanism can vary. What cannot vary is the need to delete with the current ownership token.
Delete must coordinate with renewal
The delete operation used to be the simple end of the story. Handler succeeded, so delete the message.
With renewal, delete depends on the current ownership token. If renewal has produced a newer pop receipt, deleting with an older one can fail.
That means the pump has to coordinate completion with the renewal loop:
- Stop or cancel the renewer.
- Observe whether renewal produced a newer receipt.
- Delete using the latest known receipt.
- Treat delete failure as a possible ownership problem, not just a transient broker error.
If the pump stops the renewer and awaits it before deleting, this coordination is manageable. The important part is to make the ownership token part of the completion protocol instead of treating delete as an isolated broker call.
If delete still fails because the lease moved on, the message may be processed again. That does not always mean the handler failed. It can mean the pump no longer owns the message.
That is why idempotent handlers still matter. Lease renewal reduces duplicate processing. It does not remove the possibility.
Renewal failure is a policy decision
Renewal can fail. The network can be unavailable. The queue service can throttle requests. The process can be paused long enough to miss the renewal window. A renewal call can arrive after the lease has already expired.
The naive implementation logs the renewal failure and lets the handler continue. Sometimes that is the right choice. If the handler is close to completion and its work is idempotent, finishing may be acceptable.
Sometimes it is not. If the handler is about to charge a credit card, send an email, or trigger work in another system, continuing after the pump can no longer guarantee ownership may increase duplicate side effects.
The pump needs a policy: may processing continue after renewal can no longer be guaranteed?
There is no universal answer. The choice depends on the handler’s side effects, idempotency, downstream systems, and the cost of duplicate work. What matters is that the policy is explicit. A renewal failure is not just a log line. It changes the ownership status of the message.
Cancellation joins the protocol
The sample receiver creates a linked cancellation token source for each received message. That token coordinates the handler and the renewer. The sample starts both tasks without awaiting them immediately so the two activities can run side by side:
var coordinationTokenSource =
CancellationTokenSource.CreateLinkedTokenSource(cancellationToken);
var coordinationToken = coordinationTokenSource.Token;
_ = RenewLease(message, coordinationTokenSource, coordinationToken);
_ = HandleMessage(message, coordinationTokenSource, coordinationToken);
When the handler finishes, it cancels and disposes the coordination token source in a finally block. That tells the renewer to stop:
finally
{
await coordinationTokenSource.CancelAsync();
coordinationTokenSource.Dispose();
concurrencyLimiter.Release();
}
This is not just cleanup. It is coordination. The pump must stop renewing after the message has been deleted or abandoned. It must not let a background renewal task continue after the handler has finished. It must also avoid disposing shared coordination state while another task is still trying to use it.
In production code, that usually also means keeping a reference to the renewal task and awaiting it, or otherwise making sure it has observed cancellation before disposing shared state.
Once renewal exists, cancellation stops being a single process-level signal. There is shutdown cancellation, receive-loop cancellation, handler cancellation, and per-message renewal cancellation. Those signals are related, but they do not all mean the same thing.
Graceful shutdown gets another moving part
In the bounded-concurrency pump, graceful shutdown meant stop reading and wait until the in-flight handlers returned their semaphore slots.
Lease renewal adds another responsibility. During shutdown, the pump has to decide what happens to messages that are already being handled.
The sample tries to make a canceled message visible again:
catch (OperationCanceledException)
{
await receiver.UpdateMessageAsync(
message.MessageId,
message.PopReceipt,
visibilityTimeout: TimeSpan.Zero,
cancellationToken: CancellationToken.None);
}
That is a reasonable demonstration of intent: if the pump stops before finishing the message, make the message available for another receiver quickly instead of waiting for the full lease to expire.
Production code needs to be careful here too. If a renewal already produced a newer pop receipt, making the message visible with an older receipt may fail. The same “current pop receipt” rule applies to abandon-style updates as it does to delete.
This only works while the pump still has a valid pop receipt. Once the lease has expired or another receiver has acquired the message, the pump no longer owns the message and cannot reliably change its visibility.
The shutdown path uses the same lease ownership rules as the normal completion path.
The pump now runs several protocols at once
The first receive loop had one visible activity. With lease renewal, a single in-flight message now has several concurrent activities that must agree on the outcome.
The handler decides whether business work completed. The renewer protects temporary ownership. Shutdown asks both to stop. Delete needs the latest pop receipt. The concurrency limiter needs every path to release its slot exactly once.
Each responsibility is small on its own. Together, they change the shape of the pump.
The pump is no longer only a loop around business code. It is a coordinator between the broker, the handler, the process lifetime, the concurrency limit, and the ownership token required to remove the message safely.
You can see the problem with a small experiment. Set the visibility timeout to 10 seconds, make the handler take 12 seconds, and run more than one receiver. Without renewal, the same message can become visible while the first handler is still running. With renewal, the duplicate is less likely, but the pump now has to coordinate all the renewal, cancellation, and delete behavior described above.
The complexity is not Azure Queue Storage
It would be easy to blame the queue API here. The pop receipt changes. The visibility timeout must be renewed. Delete requires the current token.
But those details exist because the queue has to support failure recovery. If a consumer receives a message and disappears, the broker needs a way to let another consumer try again. A lease is one practical way to do that.
The complexity comes from building a production-grade message pump around that lease.
Once handlers can outlive the visibility timeout, the pump must schedule renewals, keep renewal independent of business work, track mutable ownership tokens, coordinate renewal with completion, decide what renewal failure means, support cancellation, and shut down without leaving background work behind.
None of that makes lease renewal impossible. It means lease renewal is infrastructure. It needs the same care as concurrency limits, transaction boundaries, retries, and error handling.
This is why production-grade messaging frameworks spend so much engineering effort inside the pump. The first loop is easy to write. The behavior around the loop is where most of the work lives.
The same pattern shows up outside Azure Queue Storage. Azure Service Bus has a lock duration on queues and topic subscriptions. A receiver gets a message under a temporary lock, and if that lock is not renewed before it expires, the broker can make the message available to another receiver. Completing or abandoning the message after the lock has expired fails. The .NET client can renew locks automatically through MaxAutoLockRenewalDuration, but that still leaves the pump responsible for configuring the renewal window and handling lock loss.
Amazon SQS has a similar concept with its visibility timeout: receiving a message hides it for a limited time, and a long-running consumer has to extend that timeout if it wants to keep the message from being delivered again before processing finishes.
Further reading:
