Service API: Batching

What if you need to transfer large number of items to remote service? The goal is to transfer information as fast as we can. What would be the most efficient solution?

Well, the first naive approach obviously doesn't work:

AccountChange[] accountChanges = ...

foreach (var change in accountChanges)
{
    client.SendChange(change);
}

We accumulating network latencies and the overall timing is awful:

Naive
======================
Elapsed: 00:00:30.4708671
Rate: 3281.82324683501

Next popular solution is to pack all items into single batch:

AccountChange[] accountChanges = ...

client.SendChange(accountChanges);

Which gives much better timing:

OneBatch
======================
Elapsed: 00:00:02.4503693
Rate: 40810.1750213733

Could be even better? Let's analyze what happens behind the scene.
First, we need to serialize _all_ items, then we transfer bits to our remote service, then service should deserialize _all_ items, before it can start processing them.
All these steps run on single thread no matter how many idle cores you have. So using single batch we removed all network round-trips, but we added latency and it doesn't scale out.

Now the winning strategy is clear: pack items into multiple batches. Smaller batch sizes lower latency, but increase overall time. Choose carefully depending on your case.

MultiBatch
======================
Elapsed: 00:00:01.0337523
Rate: 96734.9721978853

The sample is available here

But what if last approach doesn't work for us? If, for example, sender is limited in number of connections it can use or it should send items in single atomic batch? Could we still perform better then one single batch approach?

To be continued...

Service API: Throughput and Latency

There are at least two things you should consider when designing scalable Service APIs - throughput and latency. Throughput is the number of request service can process per, for example, second. Latency is time to process single request. It is obvious, thank you, Cap! But how these two are related to scalability?

Well, throughput can be theoretically improved by scaling out your service. Add more instances and your throughput grows. I put "theoretically" because often your multiple instances end up waiting each other on some shared resource, database for example. That's why it is shouted on every corner that scalable architecture should be put into product from the beginning.

Latency is another story. You can't reduce latency by scaling out. Of course if your system is overloaded then latency goes high, you add instances and latency drops down. But you can't put it below some limit no matter how many instances you add. That's what I'm talking about!

So morale of the post: do not add latency "by design" - you can't mitigate this later on by scaling out.

But what about Service APIs? How can we add latency "by design" in Service APIs?
To be continued...