Batching and linger: throughput vs latency, an interactive simulator
Most writes pay a fixed cost per request, a network round trip, a disk flush, a commit, on top of a small cost per item. Batching sends many items in one request so the fixed cost is shared, which can raise throughput by orders of magnitude. The questions are how big a batch may get and whether to linger, holding items back to wait for more. Get them wrong and you either run out of capacity or add latency for nothing.
The playground has a producer whose requests cost 2 ms each plus 0.02 ms per item, with one request in flight at a time. Change the load, the largest batch and the linger, and watch p99 latency.
Items arrive at 3,000 a second. Each request costs 2 ms of round trip plus 0.02 ms per item, and only one is in flight at a time, so sending items one by one tops out near 495 a second. With batches of up to 100 and no linger, requests carry 6.6 items on average, 459 requests a second, at a p99 of 4.4 ms. Without any linger, batches still form on their own: whatever arrives while one request is in flight goes in the next, so batches grow exactly as fast as the load does.
Why batches help
One item per request caps the producer near 500 items a second: every item pays the full 2 ms. At 3,000 items a second the queue grows without limit and latency climbs past a second. Allowing batches of up to 100 raises the ceiling to 25,000 a second, and at 3,000 a second the p99 is 4.4 ms. Larger batches also compress better and mean fewer requests for the server to handle, often the bigger saving.
Linger: when waiting helps and when it hurts
Batching does not need a timer. With no linger, the sender sends whatever has piled up whenever it is free; items that arrive while a request is in flight form the next batch. Batches grow with the load by themselves: 1.2 items on average at 300 a second, 6.6 at 3,000, 68 at 20,000. This is the same idea as group commit in databases.
A linger holds the first item back in the hope that others arrive. At light load it only adds delay: a 50 ms linger raises the p99 from about 4 ms to 52 ms at 300 items a second. Under load it buys fuller batches and fewer requests: at 3,000 a second, a 5 ms linger cuts requests from 459 a second to 186, for about 3 ms of extra latency. That is worth it when the receiver is the bottleneck or charges per request, as with Kafka brokers, DynamoDB or an API with request quotas.
- Cap batches by count and bytes, so one batch cannot exceed request size limits or hold up everything behind it.
- Handle partial failure: a batch can half succeed; report and retry per item, idempotently.
- Batch reads too: loaders like DataLoader collect lookups made in the same tick into one query.
Batching in a Kafka producer and a database writer
# Kafka producer: batches form per partition
batch.size=65536 # bytes: the largest batch
linger.ms=5 # wait up to 5 ms for a batch to fill (default 5 in Kafka 4.0, was 0)
compression.type=zstd # bigger batches also compress better
# A database writer doing the same by hand
def writer(queue, db, max_batch=500, linger_s=0.0):
while True:
rows = [queue.get()] # block for the first row
deadline = time.monotonic() + linger_s
while len(rows) < max_batch:
try:
rows.append(queue.get(timeout=max(0, deadline - time.monotonic())))
except Empty:
break # nothing more waiting: send what we have
db.executemany("INSERT INTO events VALUES (%s, %s, %s)", rows) # one round trip, one commitIn a system design interview
Batching is how you make write-heavy paths cheap: “Click events are buffered in each API server and flushed to Kafka in batches of up to 64 KB or every 10 ms, so a broker handles thousands of requests a second instead of millions. The consumer writes to the warehouse in batches of 10,000 rows. Batching adds at most tens of milliseconds, which analytics does not mind.” Expect the follow-ups: what happens to a buffered batch when the server crashes, and how batching interacts with ordering and retries.
Read the guide to bulk imports and exports or Kafka internals, see queueing and Little’s law for why the queue explodes past capacity, or watch a strong candidate design an ad click aggregator.
Questions people ask
Why does batching increase throughput?
Because most requests carry a fixed cost, such as a network round trip, a disk flush or a commit, that does not depend on how many items they contain. Sending many items per request spreads that cost across them, so the same sender can move far more items per second.
What is linger.ms in Kafka?
The time a Kafka producer waits for more records before sending a partially full batch. A small linger, a few milliseconds, produces fuller batches and fewer requests to the brokers at the cost of a little latency; at low traffic it mostly adds delay.
Does batching always add latency?
No. Without any linger, a sender that only has one request in flight naturally batches whatever accumulated while it was busy, so batches grow with the load and items never wait longer than they would in a queue anyway. Latency is only added when the sender deliberately waits.
How big should a batch be?
Big enough that the fixed cost per request is small relative to the per-item cost at peak load, and bounded by request size limits, memory and how much work you are willing to retry if a batch fails. Cap it by both item count and bytes.
What is group commit?
A database technique where transactions committing at about the same time share one flush of the write-ahead log to disk. It is batching applied to durability: each transaction still waits for the flush, but the expensive fsync is paid once for many of them.
How do you handle failures in a batch?
A batch can partly succeed, so APIs that accept batches should report results per item, and clients should retry only the failed items with idempotency keys so that a retried batch does not apply items twice.
When should you avoid batching?
When each item must be acknowledged with the lowest possible latency and load is light, or when items must be processed strictly one at a time. Even then, letting items that arrive during an in-flight request ride in the next one usually costs nothing.