Coordinated omission in load testing: closed-loop vs open-loop latency, an interactive simulator
Load tests are how teams decide whether a service meets its latency target, and many of them hide exactly the slow moments they should catch. A closed-loop tester waits for each response before sending the next request, so when the server stalls, the tester stalls with it and stops sending. It records one slow request instead of the hundreds that real users would have sent. Gil Tene named this coordinated omission: the tester coordinates with the server to omit the bad measurements.
The playground load tests a server that answers in about a millisecond but stalls for 2 s and then for 1 s during a one-minute run at 100 requests a second.
The tester means to send 100 requests a second, but it waits for each answer before sending the next. During each stall it is stuck on one request, so it records one slow sample per stall and quietly skips the 300 requests it should have sent. The report says p99 1.4 ms and p99.9 1.4 ms: the stalls have all but vanished from the percentiles.
What the measurements show
- The closed-loop report looks perfect. p99 and p99.9 are both 1.4 ms. The two stalls appear as two slow samples among 5,700, and the 300 requests that should have been sent during them were never sent, so they never count.
- Timing from the intended send time fixes it. Counting every request the schedule called for and measuring each from when it should have gone out gives a p99 of 1.41 s, the correction tools such as wrk2 and HdrHistogram apply.
- An open-loop test sees what users see. Users do not wait for each other: everyone who arrives during a stall waits for it to end. 311 of 6,000 requests take over 100 ms and the p99 is 1.47 s, about a thousand times the closed-loop figure.
Choosing and configuring tools
# closed loop: 10 connections, each sends its next request only after the last answer
wrk -t2 -c10 -d60s http://api/health
# open loop at a constant rate, latency measured from the intended send time
wrk2 -t2 -c10 -d60s -R100 http://api/health
k6: executor: 'constant-arrival-rate', rate: 100, timeUnit: '1s'
vegeta attack -rate=100/s -duration=60s
# when recording latency yourself, record from when the request should have started
histogram.recordValueWithExpectedInterval(latency, expectedIntervalMs) // HdrHistogramThe same trap exists in production metrics: a client that measures only requests it managed to send, or a server that measures from when it started work rather than when the request arrived, hides queueing time. Measure at the edge, from arrival, and keep the maximum as well as the percentiles.
In a system design interview
When you talk about latency targets and how to verify them, mention how the load is generated: “We load test with an open-loop generator at the target request rate, so a pause in the service shows up as the queue it would cause for real users, and we record latency from the intended send time in an HDR histogram. Percentiles from a closed-loop tester would understate exactly the stalls we care about.” Expect the follow-ups: how many requests a p99.9 needs to be meaningful, and how to load test a dependency-heavy service realistically.
Read the guide to tail latency or SLOs and error budgets, see tail latency at fan-out and queueing and Little’s law, or watch a strong candidate design a metrics system.
Questions people ask
What is coordinated omission?
A measurement error in which a load generator or client waits for each response before sending the next request, so when the system stalls it stops sending and records only one slow request instead of all the requests that would have been delayed. The resulting percentiles hide the stalls.
What is the difference between closed-loop and open-loop load testing?
A closed-loop test has a fixed number of clients that each send a new request only after the previous one completes, so load drops when the system slows down. An open-loop test sends requests at a fixed rate regardless of responses, like independent users, so slowdowns create queues that show in the latency.
How do you correct for coordinated omission?
Measure each request from when it was supposed to be sent rather than when it was sent, and account for the requests that should have been sent while the client was blocked. wrk2 does this, and HdrHistogram can back-fill the missing samples given the expected interval.
Which load testing tools avoid coordinated omission?
Tools with constant-arrival-rate or open-loop modes, such as wrk2, k6 with the constant-arrival-rate executor, Gatling open injection profiles and vegeta. Tools that run a fixed number of virtual users in a loop are closed loop unless configured otherwise.
Why does coordinated omission matter for SLOs?
Because it hides precisely the pauses, such as garbage collection, failovers and lock contention, that dominate tail latency. A service can pass a p99 target in a closed-loop test and badly miss it for real users.
Can production monitoring suffer from coordinated omission?
Yes: a server that measures from when it began processing rather than when the request arrived omits queueing time, and a client that records only requests it managed to send omits the ones it held back. Measuring at the edge from arrival avoids it.
How many requests do you need to measure a p99.9?
At least several thousand, and preferably tens of thousands, so that the 0.1% tail contains enough samples to be stable. With few samples, tail percentiles are dominated by noise or by one event.