Multiple message receivers with Azure Service Bus

In the first post of this series we looked at a simple message receive loop. We tried to optimize the receive loop by moving out the message completion into a dedicated background operation. It turns out we can do more to boost the message throughput. Here is how our simple loop looked like

var receiveClient = QueueClient.CreateFromConnectionString(connectionString, queueName, ReceiveMode.PeekLock);
receiveClient.OnMessageAsync(async message =>
{
...
},
..)

We created a queue client with QueueClient.CreateFromConnection. If we carefully read the best practices for improvements using Service Bus Messaging we can see that there are multiple ways of creating queue clients or message receivers. We can also create receivers with a MessagingFactory.

var factory = await MessagingFactory.CreateAsync(address, settings).ConfigureAwait(false);
var receiveClient = await factory.CreateMessageReceiverAsync(queueName, ReceiveMode.PeekLock).ConfigureAwait(false);
receiveClient.OnMessageAsync(async message =>
{
...
},
..)

So far we haven’t achieved much. We’ve just changed the creation of the receiver. But we could now try to create multiple receive clients like the following snippets illustrate.

var factory = await MessagingFactory.CreateAsync(address, settings).ConfigureAwait(false);
var receiver1 = await factory.CreateMessageReceiverAsync(queueName, ReceiveMode.PeekLock).ConfigureAwait(false);
var receiver2 = await factory.CreateMessageReceiverAsync(queueName, ReceiveMode.PeekLock).ConfigureAwait(false);
receiver1.OnMessageAsync(async message => { ... }, ..);
receiver2.OnMessageAsync(async message => { ... }, ..);

To avoid duplication of the message receive loop logic, we would need to move the logic out into a dedicated method.

receiver1.OnMessageAsync(ReceiveMessage, ..);
receiver2.OnMessageAsync(ReceiveMessage, ..);

static async Task ReceiveMessage(BrokeredMessage message) {
   ...
}

Unfortunately, this will not bring us the desired throughput. The performance as mentioned earlier best practice documentation states:

all clients (senders in addition to receivers) that are created by the same factory share one TCP connection. The maximum message throughput is limited by the number of operations that can go through this TCP connection. The throughput that can be obtained with a single factory varies greatly with TCP round-trip times and message size. To obtain higher throughput rates, you should use multiple messaging factories.

So ideally we would need to have a one-to-one relationship between factories and receivers. Let’s apply that to your snippet above.

var factory1 = await MessagingFactory.CreateAsync(address, settings).ConfigureAwait(false);
var factory2 = await MessagingFactory.CreateAsync(address, settings).ConfigureAwait(false);
var receiver1 = await factory1.CreateMessageReceiverAsync(queueName, ReceiveMode.PeekLock).ConfigureAwait(false);
var receiver2 = await factory2.CreateMessageReceiverAsync(queueName, ReceiveMode.PeekLock).ConfigureAwait(false);
receiver1.OnMessageAsync(ReceiveMessage, ..);
receiver2.OnMessageAsync(ReceiveMessage, ..);

With this approach, each receiver is created by a dedicated messaging factory. There is no TCP connection sharing involved anymore, and each receiver has a dedicated connection. The concurrency settings can be applied as desired to each receiver. With this approach, we can have as many factories and receivers as needed and concurrently fetch messages from the same queue on multiple dedicated connections. If we combine this with a wisely chosen PrefetchCount and potentially partitioned queues or topics we can speed up our message, receive loop to race car speed.

How this influences the batch completion will be outlined in the next installment.