Home / Articles posted by Daniel Marbach

Introduction to the MultiProducerConcurrentConsumer for Azure Service Bus message completion

In the last post in the series about Azure Service Bus message completion, I briefly talked how the message batch completion could benefit from a more reactive approach. Before I dive into the inner workings of the reactive structure that I’m going to outline in the next few posts, I need to briefly recap the functional and non-functional requirements we need from the component.

The component shall

  • Make sure messages are only completed on the receiver they came from
  • Reduce the number of threads used when the number of clients is increased
  • Autoscale up under heavy load
  • Scale down under light load
  • Minimise the contention on the underlying collections used
  • Be completely asynchronous
  • Implements a push based model from the producer and consumer perspective
  • Respect the maximum batch sized defined by the client of the component or a predefined push interval
  • Provide FIFO semantics instead of LIFO
  • Be as lock-free as possible

From a design viewpoint the MultiProducerConcurrentConsumer (MPCC) component will look like

The MPCC manages internally N number of slots that can contain items of type TItem. Whenever the timeout or the batch size is reached the component fans out up to a provided concurrency per slot the completion callback.

class MultiProducerConcurrentConsumer<TItem> {
    public MultiProducerConcurrentConsumer(int batchSize, TimeSpan pushInterval, int maxConcurrency, int numberOfSlots) { }

    public void Start(Func<List<TItem>, int, object, CancellationToken, Task> pump) { }

    public void Start(Func<List<TItem>, int, object, CancellationToken, Task> pump, object state) { }

    public void Push(TItem item, int slotNumber) { }

    public async Task Complete(bool drain = true) { }
}    

Shown above is the class outline that we are going to use for the component. The constructor allows specifying the batch size, the push interval, the number of slots and the concurrency per slot. The component has a method to push items to it called Push. As soon as the component is started the provided pump delegate will be called whenever the batch size is reached or the push interval. The concurrency per slot is probably best explained by giving an example.

Let’s assume we have a batch size of 500 and a push interval of 2 seconds. When producers can produce 1500 items per second the batch size is reached three times per second. MPCC will auto-scale in that case up to the provided concurrency. So in the above example, it will take out 500 items and invoke the callback and then concurrently try to fetch another 500 items and again another 500 items until the slot is empty or the maximum concurrency is reached.

The Complete method is available for the deterministic stopping of the MPCC. It allows to asynchronously wait for all slots to be drained (by default) or immediately stop and throw away all items that haven’t been processed.

In the next post, we are going to depict the inner workings of the MultiProducerConcurrentConsumer, hope you’ll enjoy.

Complete messages where they came from with Azure Service Bus

In the last post, we made a simplistic attempt to speed up batch completion of messages by having multiple dedicated background completion tasks. Unfortunately, this introduced an interesting side effect.


static async Task BatchCompletionLoop() {
   while(!token.IsCancellationRequested) {
         ...
         await receiveClient.CompleteBatchAsync(lockTokens).ConfigureAwait(false);
         ...
      }
      await Task.Delay(TimeSpan.FromSeconds(5), token).ConfigureAwait(false);
   }
}

The code above is using a single receive client to complete message lock tokens in batches. Those lock token could come from multiple receive clients, but they would end up being completed on the same instance. Depending on the transport type you use it might work or it won’t. When you are using the NetMessaging (also known as SBMP) transport type the above code will just work. If you switch to AMQP as the transport type, the code will blow up with the following exception

Microsoft.ServiceBus.Messaging.MessageLockLostException: The lock supplied is invalid. Either the lock expired, or the message has already been removed from the queue.
   at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
   at Microsoft.ServiceBus.Messaging.MessageReceiver.EndCompleteBatch(IAsyncResult result)

For Azure Service Bus v3 and higher AMQP is the new standard protocol and SMBP is an opt-in choice.

The proprietary SBMP protocol that is also supported is being phased out in favor of AMQP. see documentation

What does that mean for our message completion? To have a robust completion code which works for both transport types, we have to make sure we only complete lock tokens on receivers the messages were coming from.

In its simplest form, we would need the following moving pieces. A concurrent stack or queue per message receivers, a generic message handling body which closes over the concurrent stack as an input parameter and similarly a parameterized completion loop.

var lockTokensToComplete = new ConcurrentStack<Guid>[numberOfReceivers];
// initialize the concurrent stacks
  
receiveClient1.OnMessageAsync(message => ReceiveMessage(message, lockTokensToComplete[0]);
...
receiveClientN.OnMessageAsync(message => ReceiveMessage(message, lockTokensToComplete[N-1]);

static async Task ReceiveMessage(BrokeredMessage message, ConcurrentStack<Guid> lockTokensToComplete) {
   await DoSomethingWithTheMessageAsync().ConfigureAwait(false);
   lockTokensToComplete.Push(message.LockToken);
}

and the completion loop

for(int i = 0; i < numberOfReceivers; i++) { 
   completionTasks[i] = Task.Run(() => BatchCompletionLoop(receivers[i], lockTokensToComplete[i]));
}
 
static async Task BatchCompletionLoop(MessageReceiver receiver, ConcurrentStack<Guid> lockTokensToComplete) {
   while(!token.IsCancellationRequested) {
    var lockTokens = new Guid[100];
      int numberOfItems = lockTokensToComplete.TryPopRange(lockTokens)
      if(numberOfItems > 0) {
         await receiver.CompleteBatchAsync(lockTokens).ConfigureAwait(false);
      }
      await Task.Delay(TimeSpan.FromSeconds(5), token).ConfigureAwait(false);
   }
}

We have reduced the contention since we have a dedicated lock token concurrent collection to operate on per receiver. We have a dedicated completion loop which is self-contained and only operates on the correct receiver and lock token collection pair. So far so good. Despite that we are still wasting a lot of unnecessary resources. We acquired up to the number receivers dedicated completion background tasks which might idle. And we still don’t exactly know if this code constructs scales elastically up and down depending on the number of messages received and the number of clients.

Ideally, we would have a reactive approach which automatically adapts based on parameters we give it. Building such a reactive approach will be the focus of the next few posts. Stay reactive!

Batch completion with multiple receivers on Azure Service Bus

In the last post, we created multiple receivers with multiple factories to speed up our message processing. Unfortunately, this has some consequences for our background completion loop. Since the message processing logic is shared between multiple receivers, all the receivers will try to push lock tokens into the concurrent stack. So the following code will be called by numberOfReceivers * concurrencyPerReceiver concurrently.

// same as before
await DoSomethingWithTheMessageAsync().ConfigureAwait(false);
lockTokensToComplete.Push(message.LockToken);
// same as before

So for example when we’d use 10 receivers with each a concurrency setting of 32 we’d be ending up pushing lock tokens to the concurrent stack from up to 320 simultaneous operations. Not a big deal you could say since the ConcurrentStack implementation is lock-free. Unfortunately, lock-free doesn’t necessarily mean it is contention free.

ConcurrentQueue<T> and ConcurrentStack<T> are completely lock-free in this way. They will never take a lock, but they may end up spinning and retrying an operation when faced with contention (when the CAS operations fail). Read more

The concurrent stack implementation internally uses a technique called spinning and retrying (for more information see the excellent Blocking vs. Spinning section from Joe Albahari). Under contention with a large number of concurrent operations these operations can fail multiple times and might become less efficient than we assumed them to be.

In the previous post I made the following statement:

When we’d received several hundred messages per seconds our randomly chosen “complete every one-hundredth messages” and then “sleep for five seconds” might turn out to be a suboptimal choice.

With multiple receivers pushing lock tokens into the concurrent stack, we might have made our problem even worse. It is entirely possible that our multiple concurrent receivers can fill the concurrent stack faster with lock tokens than our completion loop manage to complete. Our previously chosen five seconds sleep duration might turn out to be way too long under heavy load.

How about we just spin up multiple batch completion tasks like the following?

var completionTasks = new Task[numberOfReceivers];

for(int i = 0; i < numberOfReceivers; i++) { 
   completionTasks[i] = Task.Run(() => BatchCompletionLoop());
}

static async Task BatchCompletionLoop() {
   while(!token.IsCancellationRequested) {
    var lockTokens = new Guid[100];
      int numberOfItems = lockTokensToComplete.TryPopRange(lockTokens)
      if(numberOfItems > 0) {
         await receiveClient.CompleteBatchAsync(lockTokens).ConfigureAwait(false);
      }
      await Task.Delay(TimeSpan.FromSeconds(5), token).ConfigureAwait(false);
   }
}

Now we’ve just made the contention problem on the concurrent stack even worse. Multiple background completion operations are competing on the concurrent stack as well. Because all those completion tasks would have the same Task.Delay value, there is a high chance we would see a pattern where multiple background completion operations are dispatch on a worker thread but only a few would succeed and idle again, effectively wasting a lot of resources in production.

Leaving contention and resource waste aside we’ve now introduced another major flaw in our completion logic. Can you spot it? If not, don’t worry. I’ll pick it up in the next post.

Multiple message receivers with Azure Service Bus

In the first post of this series we looked at a simple message receive loop. We tried to optimize the receive loop by moving out the message completion into a dedicated background operation. It turns out we can do more to boost the message throughput. Here is how our simple loop looked like

var receiveClient = QueueClient.CreateFromConnectionString(connectionString, queueName, ReceiveMode.PeekLock);
receiveClient.OnMessageAsync(async message =>
{
...
},
..)

We created a queue client with QueueClient.CreateFromConnection. If we carefully read the best practices for improvements using Service Bus Messaging we can see that there are multiple ways of creating queue clients or message receivers. We can also create receivers with a MessagingFactory.

var factory = await MessagingFactory.CreateAsync(address, settings).ConfigureAwait(false);
var receiveClient = await factory.CreateMessageReceiverAsync(queueName, ReceiveMode.PeekLock).ConfigureAwait(false);
receiveClient.OnMessageAsync(async message =>
{
...
},
..)

So far we haven’t achieved much. We’ve just changed the creation of the receiver. But we could now try to create multiple receive clients like the following snippets illustrate.

var factory = await MessagingFactory.CreateAsync(address, settings).ConfigureAwait(false);
var receiver1 = await factory.CreateMessageReceiverAsync(queueName, ReceiveMode.PeekLock).ConfigureAwait(false);
var receiver2 = await factory.CreateMessageReceiverAsync(queueName, ReceiveMode.PeekLock).ConfigureAwait(false);
receiver1.OnMessageAsync(async message => { ... }, ..);
receiver2.OnMessageAsync(async message => { ... }, ..);

To avoid duplication of the message receive loop logic, we would need to move the logic out into a dedicated method.

receiver1.OnMessageAsync(ReceiveMessage, ..);
receiver2.OnMessageAsync(ReceiveMessage, ..);

static async Task ReceiveMessage(BrokeredMessage message) {
   ...
}

Unfortunately, this will not bring us the desired throughput. The performance as mentioned earlier best practice documentation states:

all clients (senders in addition to receivers) that are created by the same factory share one TCP connection. The maximum message throughput is limited by the number of operations that can go through this TCP connection. The throughput that can be obtained with a single factory varies greatly with TCP round-trip times and message size. To obtain higher throughput rates, you should use multiple messaging factories.

So ideally we would need to have a one-to-one relationship between factories and receivers. Let’s apply that to your snippet above.

var factory1 = await MessagingFactory.CreateAsync(address, settings).ConfigureAwait(false);
var factory2 = await MessagingFactory.CreateAsync(address, settings).ConfigureAwait(false);
var receiver1 = await factory1.CreateMessageReceiverAsync(queueName, ReceiveMode.PeekLock).ConfigureAwait(false);
var receiver2 = await factory2.CreateMessageReceiverAsync(queueName, ReceiveMode.PeekLock).ConfigureAwait(false);
receiver1.OnMessageAsync(ReceiveMessage, ..);
receiver2.OnMessageAsync(ReceiveMessage, ..);

With this approach, each receiver is created by a dedicated messaging factory. There is no TCP connection sharing involved anymore, and each receiver has a dedicated connection. The concurrency settings can be applied as desired to each receiver. With this approach, we can have as many factories and receivers as needed and concurrently fetch messages from the same queue on multiple dedicated connections. If we combine this with a wisely chosen PrefetchCount and potentially partitioned queues or topics we can speed up our message, receive loop to race car speed.

How this influences the batch completion will be outlined in the next installment.

Async method without cancellation support, do it my way.

In the last post, I talked about Dennis Doomen’s LiquidProjects project and the challenge they faced with asynchronous APIs that were not cancelable. Dennis came up with the following solution to the infinitely running Task.Delay operations.

public static class TaskExtensions {
    public static async Task<TResult> WithWaitCancellation<TResult>(this Task<TResult> task,
            CancellationToken cancellationToken) {
        using (var combined = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken))            
        {
            var delayTask = Task.Delay(Timeout.Infinite, combined.Token);
            Task completedTask = await Task.WhenAny(task, delayTask);
            if (completedTask == task)
            {
                combined.Cancel();
                return await task;
            }
            else
            {
                cancellationToken.ThrowIfCancellationRequested();
                throw new InvalidOperationException("Infinite delay task completed.");
            }
        }
    }
}

The code above creates a linked token source, an infinitely delayed task which observes the token referenced by the linked token source. When the outer token source cancels the linked token source will also cancel the token owned by it. When the task returned by Task.WhenAny is the actual work task then the linked token source is canceled. This will automatically cancel the delayed task. The benefit of this implementation is its simplicity. The runtime performance is good but we can do better especially when we’d be using the extension method over and over again we can do a few optimizations. Here is a possible approach

public static Task<TResult> WaitWithCancellation<TResult>(this Task<TResult> task, CancellationToken token = default(CancellationToken))
{
    var tcs = new TaskCompletionSource<TResult>();
    var registration = token.Register(s => {
        var source = (TaskCompletionSource<TResult>) s;
        source.TrySetCanceled();
    }, tcs);

    task.ContinueWith((t, s) => {
        var tcsAndRegistration = (Tuple<TaskCompletionSource<TResult>, CancellationTokenRegistration>) s;

        if (t.IsFaulted && t.Exception!= null) {
            tcsAndRegistration.Item1.TrySetException(t.Exception.GetBaseException());
        }

        if (t.IsCanceled) {
            tcsAndRegistration.Item1.TrySetCanceled();
        }

        if (t.IsCompleted) {
            tcsAndRegistration.Item1.TrySetResult(t.Result);
        }

        tcsAndRegistration.Item2.Dispose();
    }, Tuple.Create(tcs, registration), CancellationToken.None, TaskContinuationOptions.ExecuteSynchronously, TaskScheduler.Default);

    return tcs.Task;
}

Instead of using a delayed task with an infinite time span we use a TaskCompletionSource. The TaskCompletionSource represents a task which is only completed, canceled or faulted when we instruct the source accordingly. So essentially this is an infinitely running task. On the token passed in we register a registration which will get triggered when the token gets canceled. When the token gets canceled, we set the completion source to canceled as well. On the actual task, we register a continuation which sets the state of the task completion source according to the state of the antecedent task (which is the actual worker task). Inside the continuation, we also dispose the token registration not to leak memory. The continuation is scheduled with a runtime hint ExecuteSynchronously to tell the TPL runtime that the work running inside the continuation is short lived and can try to be executed while completing the antecedent task.

To avoid closure allocations we use for the token registration and the continuation the overloads which allow us to pass in a state object. Where we need to pass in multiple things into the delegate we use a tuple to represent those items. Furthermore, we use the TrySet methods to set the task completion source in a safe manner into its final state to avoid raising exceptions that could occur due to races.

Let’s see what the runtime differences of both code snippets are:

You can try it yourself with BenchmarkDotNet, and the tests found on my MicroBenchmark repo. So the second snippet is more complex to understand but it is overall faster and slightly better because it allocates less garbage and omits the asynchronous state machine generation.

Please don’t get me wrong this is slightly esoteric. Depending on your application or system you are building, I would usually go for Dennis’ approach because it is simpler to understand and good enough. If you need a bit more performance and want to save allocations the second approach written by me might come in handy.

Thanks again Dennis for this cool asynchronous challenge.

Improved batch completion loop for Azure Service Bus

In the last post, we created a dedicated completion loop inside a worker thread to complete messages in batches. There are a few obvious or not so obvious things we can improve in our code. But first, how did it look like again?

// same as before
var batchCompletionTask = Task.Run(async () => {
   while(!token.IsCancellationRequested) {
      var lockTokens = new Guid[100];
      int numberOfItems = lockTokensToComplete.TryPopRange(lockTokens)
      if(numberOfItems > 0) {
         await receiveClient.CompleteBatchAsync(lockTokens).ConfigureAwait(false);
      }
      await Task.Delay(TimeSpan.FromSeconds(5), token).ConfigureAwait(false);
   }
});

We could make a simple assumption: If the numberOfItems returned by TryPopRange is equal the maximum lock token range we want to complete in batches (here one hundred) then we have potentially more things to complete, and we can try to avoid the delay.

// same as before
var batchCompletionTask = Task.Run(async () => {
   while(!token.IsCancellationRequested) {
      // same as before
      if(numberOfItems == lockTokens.Length) {
         continue;
      }
      await Task.Delay(TimeSpan.FromSeconds(5), token).ConfigureAwait(false);
   }
});

So if we were able to fill the lock token array we assume we have more to fetch and continue in the while loop. If the ConcurrentStack happens to be empty after the continuation, we will then end up in the Task.Delay. If there are still lock tokens to fetch, we will continue fetching at least another round. What else could we improve?

We randomly pick a Guid array size of one hundred. An Azure Service Bus request can be up to 256 KB in size. A Guid has a size of 16 bytes. One hundred Guids, therefore would have a size of roughly 1.6 KB. We could easily increase the array size to a much larger number. For example, 5000 Guids would mean a payload size of approximately 80 KB which is roughly a third of the maximum payload size. This seems to be a reasonable trade-off. Of course, you’d need to make your analysis based on your application’s need and your Service Bus tier and data center you are connecting to.

We could also start tweaking the delay interval. Are five seconds too much for faster receiving of messages? Or should we increase the timespan for lighter load? These are all questions that are hard to answer without particular non-functional requirements of your application or system.

We have shed light on message receiving and mostly batch completion outside the receive loop. In the next installment, we will circle back to the message receive loop and see how we can get even more receive performance and how this potentially impacts our completion logic.

 

Another attempt to batch complete with Azure Service Bus

In the last post, we tried to be smart with batch completion and completed every one-hundredth message. Unfortunately, this way of doing completion might not work under with few messages in the queue. We decided to go back to the drawing board and see if we can come up with a better approach which can cope with high but also small loads of messages. The heart of the code we wrote in the last post (omitting the exception handling part) looks like

var lockTokensToComplete = new ConcurrentStack<Guid>();
 
receiveClient.OnMessageAsync(async message =>
{
   // same as before
   await DoSomethingWithTheMessageAsync().ConfigureAwait(false);
   lockTokensToComplete.Push(message.LockToken);
   // same as before
},..)

Instead of completing messages in batches of hundred lock tokens we can try to move out the batch completion into a dedicated batch completion task. Hopefully that is not too hard. Let’s see

var tokenSource = new CancellationTokenSource();
var token = tokenSource.Token;

var batchCompletionTask = Task.Run(async () => {
   while(!token.IsCancellationRequested) {
      var lockTokens = new Guid[100];
      int numberOfItems = lockTokensToComplete.TryPopRange(lockTokens)
      if(numberOfItems > 0) {
         await receiveClient.CompleteBatchAsync(lockTokens).ConfigureAwait(false);
      }
      await Task.Delay(TimeSpan.FromSeconds(5), token).ConfigureAwait(false);
   }
});

We schedule a task to be run on the worker thread pool to offload the actual completion loop away from the thread that is starting the task. We loop until the cancellation is requested on the cancellation token source which is linked to the token passed as closure into the completion loop. We then try top pop a range of lock tokens from the ConcurrentStack. If we received more than zero items, we would complete the lock tokens we popped on the receiveClient. At the end of the loop, we asynchronously sleep until either we shut down or five seconds are over.

This is beautiful and straightforward. We have a dedicated completion circuit, the contention on the concurrent stack is in a controllable range. Under small load, we complete lock tokens in in batches of one to maximum one hundred tokens. If we receive only a limited number of messages, the loop might complete messages with their tokens one by one (for example when we receive a message every 6 seconds). But what happens when the load increases?

When we’d received several hundred messages per seconds our randomly chosen “complete every one-hundredth messages” and then “sleep for five seconds” might turn out to be a suboptimal choice. We will dive more into this topic in upcoming posts. Could we try to optimize a few things in this simple loop? Feel free to exercise your brain or just wait for the next post 😉

 

Async method without cancellation support, oh my!

Sometimes you have to interact with asynchronous APIs that you don’t control. Those APIs might be executing for a very long time but have no way to cancel the request. Dennis Doomen was exactly in such a situation while building his opinionated Event Sourcing projections library for .NET called LiquidProjections.

The library is using NEventStore under the covers. The actual access to the NEventStore is adapted in a class called NEventStoreAdapter. The adapter takes care of loading pages from the store from a given checkpoint onwards. This asynchronous method is running in potentially indefinitely because of the underlying NEventStore EventStore client. Let’s look at code:

await eventStoreClient.GetPageAsync(checkpoint).ConfigureAwait(false);

So in his case, the pretty simple sample code above could execute forever, but it cannot be canceled since there is no overload available which accepts a CancellationToken. Dennis wrote the following extension to be able to write an asynchronous execution which is cancelable:

public static class TaskExtensions {
   public static async Task<TResult> WithWaitCancellation<TResult>(this Task<TResult> task,
              CancellationToken cancellationToken) {
   Task completedTask = await Task.WhenAny(task, Task.Delay(Timeout.Infinite, cancellationToken));		             
   if (completedTask == task) {		
      return await task;		
   }		
   else {		             
      cancellationToken.ThrowIfCancellationRequested();		
      throw new InvalidOperationException("Infinite delay task completed.");
   }
}

and here the usage of this extension method

await eventStoreClient.GetPageAsync(checkpoint).WithWaitCancellation(cancelationToken).ConfigureAwait(false);

The idea of the code was the following:

We package the actual task to be executed together with a Task.Delay(Timeout.Infinite) into a Task.WhenAny. When the actual task completes, it is returned from WhenAny, and we await the outcome of the task*. If the completed task is not the actual task, then we ran into the case where the Task.Delay was canceled because the cancellation token passed into that task was canceled. In this case, it is sufficient to throw the OperationCanceledException with ThrowIfCancellationRequested.

That is a nifty idea and a quite clever solution. Unfortunately, Dennis found out that this extension method produced a nasty memory leak when running for a longer period. When the actual task completes the task representing the Task.Delay operation will still be executing and therefore leak memory. Dennis came up with a nice solution with linked cancellation token sources.

His tweet

Note to self: always await or cancel a call to Task.Delay if one wants to avoid memory leaks…

caught my attention and I started working on a more robust and scalable extension method that doesn’t require Task.Delay. I will discuss the extension method I came up with in the next post. Thanks Dennis for this nice asynchronous challenge!

* Remember await a task makes the task result being materialized. If there is a result it is returned, if the task is faulted the exception is unpacked and rethrown.

Let’s try batch completion of messages on Azure Service Bus

In the last post, we deferred the message completion task into the background to remove the additional latency from the message receive callback. This time we want to save even more latency by batching multiple message completions together into a single Azure Service Bus broker call. It turns out this is possible with the Azure SDK.

The MessageReceiver has a method called CompleteBatchAsync which accepts an IEnumerable<Guid> called lockTokens. But where do we get these lock tokens from? It turns out the BrokeredMessage type that is passed into the OnMessageAsync callback contains a GUID property called LockToken. The lock token is a GUID generated by the service bus which uniquely identifies a message on the broker. So if we managed to track the lock tokens for the messages, we receive we can at a later stage execute CompleteBatchAsync. Let’s see how this would change our message receive logic.

var lockTokensToComplete = new ConcurrentStack<Guid>();
 
receiveClient.OnMessageAsync(async message =>
{
   try {
      await DoSomethingWithTheMessageAsync().ConfigureAwait(false);
      lockTokensToComplete.Push(message.LockToken);
   catch(Exception) {
      // in case of an exception make it available again immediately
      await message.AbandonAsync().ConfigureAwait(false);
   }
},..)

With this small change, we would be pushing lock tokens of messages to be completed into a ConcurrentStack. Only if a message fails we’d be directly abandoning the message asynchronously inside the message receive callback. The assumption / we are making here is that the contention on the ConcurrentStack will be far less than the latency on the remote call to complete a message. But at some point, we have to complete the messages.Can we do it directly in the callback like illustrated with the following code?


receiveClient.OnMessageAsync(async message =>
{
   // same as before
   var lockTokens = new Guid[100];
   if(lockTokensToComplete.Count > 100 && lockTokensToComplete.TryPopRange(lockTokens) > 0) {
      await receiveClient.CompleteBatchAsync(lockTokens).ConfigureAwait(false);
   }
   // same as before
},..)

We’ve chosen a concurrent stack here in order to be able to do range pops of lock tokens. This might seem a bit strange since we’d we completing messages with the lock token in another order than we received it. Normally that is not a big deal since in concurrency scenarios we should not make any assumptions about ordering.

With a lot of messages in the queue, and when the queue constantly has a particular load of messages coming in, this code would improve the latency. We would only do a remote call every hundred messages. So we theoretically save 99% of the latency we’d have when completing every message. What a success! But wait, there is a problem in this code: What if we only receive one message every ten seconds?

Assuming a peek lock duration of roughly thirty seconds, we would almost always run into the expiration of the peek lock duration and therefore the messages would be processed over and over again until they’d get dead lettered by the Azure Service Bus. That’s some bad hat harry!

So how can we make this trouble maker better? Well, I think you know it already: That’s a topic for another post.