Real-time market data distribution

How price feeds reach traders and apps: exchange feeds and multicast, sequence numbers and gap recovery, order book snapshots plus incremental updates, conflation for slow consumers, fan-out to web and mobile clients, entitlements, historical tick storage, and latency tiers.

Reading is half of it. See this used in a real interview: walk through Design a Stock Exchange →

A stock exchange or crypto platform generates a torrent of updates: every order added, changed, cancelled or filled changes the order book, and every trade changes the last price. Professional traders need every update with microsecond latency; a mobile app user needs a smooth price ticker a few times per second. Market data systems serve both from the same source. They come up as part of exchange designs and as "design a stock ticker" or "design live sports scores" questions.

The source

The matching engine emits an ordered stream of events per instrument: order book changes and trades, each with a sequence number. See low-latency systems and matching engines. Market data publishers turn this into feeds:

• Full depth (level 3): every order event.
• Aggregated depth (level 2): total quantity at each price level, top N levels.
• Top of book (level 1): best bid, best ask, last trade.

Low-latency distribution

For co-located professional clients:

• UDP multicast sends each packet once to many subscribers on the network, keeping latency equal and low regardless of the number of receivers.
• Multicast is unreliable, so every message has a sequence number; receivers detect gaps and request retransmission from a recovery service (or use a redundant A/B feed on separate network paths and take whichever packet arrives first).
• Periodic snapshots let late joiners or receivers who lost too much rebuild the book: load the snapshot, then apply incremental messages after its sequence number.

Snapshot plus incremental updates

The core pattern for any consumer:

1. Subscribe to the incremental stream and buffer it.
2. Fetch a snapshot of the book at sequence S.
3. Discard buffered updates at or below S; apply the rest in order.
4. On any gap, resync from a new snapshot.

This keeps clients consistent without resending the full book on every change. See message ordering and sequence numbers.

Conflation for slow consumers

A retail app cannot (and need not) process thousands of updates per second per symbol. Conflation merges updates: for each symbol, keep only the latest value and send it at a fixed rate (say 4 times per second) or when the consumer is ready. Slow consumers receive fewer, fresher updates instead of falling behind and building queues. This is backpressure tailored to state that supersedes itself. See load shedding and backpressure.

Fan-out to apps and browsers

• Edge distribution servers subscribe to the internal feed and push to clients over WebSockets (or SSE), with per-client subscriptions by symbol. See WebSockets vs SSE vs long polling.
• Popular symbols are subscribed by millions; distribution servers share one upstream subscription per symbol and fan out locally.
• Clients reconnect with jittered backoff and resubscribe, receiving a fresh snapshot. See the thundering herd problem.
• Regional distribution points bring data close to users. See CDN and edge.

Latency tiers and entitlements

Data is a product: exchanges charge for real-time feeds, and regulations or licences may require delayed data (15 minutes) for unpaid users. The distribution layer enforces entitlements per user and symbol, and may serve the same feed real-time to some clients and delayed to others. See authorization and permissions.

Derived data

Candles (open, high, low, close per minute), volume, indices and indicators are computed by stream processors from trades, windowed by event time. See windowing and watermarks.

Historical storage

Every tick is stored for charts, analysis, backtesting and compliance:

• Columnar, time-partitioned storage optimised for scans by symbol and time range, with heavy compression. See time-series data and compression.
• Precomputed candles at several resolutions for fast charting.

Beyond finance

The same patterns serve live sports scores, auction prices, betting odds, live leaderboards and multiplayer state: sequenced updates, snapshot plus deltas, conflation for slow clients, and massive fan-out. See Design a Leaderboard.

In the interview

For Design a Stock Exchange: "The matching engine emits sequenced book and trade events; co-located clients get multicast feeds with A/B redundancy, gap recovery and periodic snapshots; a distribution tier conflates per symbol and pushes to apps over WebSockets with entitlement checks; stream jobs build candles; all ticks land in compressed columnar storage."

Checklist

• Sequenced events per instrument from the matching engine.
• Multicast with gap recovery, redundant feeds and snapshots for low-latency clients.
• Snapshot plus incremental sync for every consumer.
• Conflation for slow consumers.
• Shared upstream subscriptions and local fan-out to WebSocket clients.
• Entitlements and delayed feeds; derived candles; historical tick storage.