Low-latency systems and matching engines

How exchanges and other microsecond systems are built: single-threaded deterministic cores, in-memory order books, sequencers and event logs, replication by replay, avoiding garbage collection and locks, kernel bypass, and fairness.

Reading is half of it. See this used in a real interview: walk through Design a Stock Exchange →

Most web systems measure latency in milliseconds. Exchanges, trading systems and some game servers measure it in microseconds, and they must also be perfectly fair and never lose an order. The architecture that achieves this looks very different from a typical microservice stack: one thread doing all the critical work, everything in memory, and a log that makes the whole thing replayable. Stock exchange questions test whether you know this design.

The order book

An order book holds resting buy orders (bids) and sell orders (asks) for one instrument, sorted by price, then time (price-time priority). A new order:

• Matches against the best opposite orders while prices cross, producing trades.
• Rests in the book with any remaining quantity (limit orders), or is cancelled (market or immediate-or-cancel orders).

Data structures: a map from price level to a queue of orders, with the best price at the top (a sorted tree or an array indexed by price ticks), plus a hash map from order id to order for fast cancels. All in memory.

Single-threaded and deterministic

The matching engine for each instrument (or group of instruments) runs on one thread:

• No locks, no contention, no race conditions: orders are processed strictly in sequence.
• Deterministic: the same input sequence always produces the same output. This is what makes replication and recovery simple.
• One core can process millions of orders per second when it never waits for I/O.

Scaling comes from partitioning by instrument, not from multithreading one book. See sharding and partitioning.

Sequencer and event log

Before matching, a sequencer assigns each incoming order a global sequence number and writes it to a durable, replicated log. The engine consumes the log in order. Consequences:

• Fairness: arrival order is defined once, by the sequencer.
• Recovery: replay the log (from a snapshot) to rebuild the exact state.
• Replication: standby engines consume the same log and stay in identical state, ready to take over within milliseconds.
• Downstream systems (market data, clearing, risk, audit) consume the output event stream.

This is event sourcing at its most extreme. See event sourcing and CQRS.

Avoiding latency spikes

At microsecond scale, the enemies are pauses and jitter:

• No garbage collection pauses: preallocate objects and reuse them (object pools), or use languages and settings without GC on the hot path.
• No locks or syscalls on the hot path; pass data between threads with lock-free ring buffers (the LMAX Disruptor pattern).
• CPU pinning: dedicate cores to critical threads, isolated from the OS scheduler.
• Cache-friendly data: arrays over pointer-chasing structures, data laid out in memory order.
• Kernel bypass networking (DPDK, specialised NICs) to skip the operating system's network stack.
• Batching the durable log writes without delaying individual acknowledgements beyond the budget.

Measure p99.9 and the maximum, not averages. See tail latency.

Around the core

• Gateways accept client connections, authenticate, validate and rate-limit orders, then forward to the sequencer.
• Risk checks (sufficient balance, position limits) happen before matching; they must also be fast, often in memory per account.
• Market data publishers broadcast book updates and trades, frequently over multicast to many subscribers at once.
• Clearing and settlement happen later, asynchronously, from the trade log. See payments and ledgers.

Fairness and colocation

Exchanges must treat participants equally: equal cable lengths in the data centre, randomised or batched processing in some venues, and deterministic sequencing. Some modern venues use frequent batch auctions (match every few milliseconds) to reduce the value of pure speed.

Beyond exchanges

The same ideas apply to other systems that need fast, consistent decisions on shared state: game servers (a single authoritative simulation loop per match), real-time bidding, matchmaking pools and in-memory leaderboards. See Design Matchmaking and Design a Leaderboard.

In the interview

For Design a Stock Exchange: gateways, then a sequencer writing to a replicated log, then a single-threaded in-memory matching engine per instrument partition, with hot standbys replaying the same log, and market data and clearing consuming the output. Then explain how you avoid pauses on the hot path.

Checklist

• In-memory order book with price-time priority and fast cancels.
• Single-threaded deterministic engine, partitioned by instrument.
• Sequencer plus durable log for fairness, recovery and replication.
• Hot standbys replaying the log.
• No GC, locks or syscalls on the hot path; pinned cores; ring buffers.
• Pre-trade risk checks, market data fan-out, asynchronous clearing.