Design Robinhood

• Live prices: Watchlists and the detail page show bid, ask and last for 10,000 symbols. A person reads a price a few times a second at most, and only the newest value matters.
• Place market and limit orders: Idempotent on a client order id, and checked against market hours, halts, buying power and shares owned before anything leaves for a venue.
• Cancel an open order: A cancel is a request, not a guarantee: the order may fill first, and the app must say so.
• Order status and fills in real time: New, sent, partially filled, filled, cancelled, rejected. Pushed over the app’s socket, and as a notification when the app is closed.
• A portfolio value that moves live: Cash plus each position times its latest price, moving with every tick, with the day’s change against yesterday’s close.
• Stop and good-til-cancelled orders: Held by the broker and triggered on the price stream, because venues do not hold a stop, and many accept only day orders.
• Price alerts: Notify once when a symbol crosses a threshold. ~10 M active alerts, heavily skewed to a few popular symbols.
• Books, records and reconciliation: An immutable audit trail of every order event for regulators, and a nightly match of trades, positions and cash against the clearing firm.
• Out of scope: Options, crypto, margin and short selling, fractional shares, bank transfers (see Design Venmo), the matching engine (see Design a Stock Exchange), and clearing and settlement themselves.

• No duplicate or lost orders (one order per tap, every fill booked): A duplicated buy spends the customer’s cash twice and cannot be taken back; a lost fill leaves positions wrong. Both are worse than a slow order.
• Order entry latency (p99 < 50 ms in our systems): From the gateway to the order leaving the FIX gateway: ~5 ms admission, ~15 ms checks and one shard transaction, ~5 ms to store and send. The phone’s network adds 50 to 200 ms we do not control.
• Quote freshness (p99 < 500 ms feed to screen): Up to 50 ms of batching, up to 250 ms of tick and the mobile network. Prices never move backwards on screen, even after a reconnect.
• Scale at the open (3 M sockets · 20 k orders/s): About 100× the day’s average order rate, arriving within seconds of 9:30. This is the requirement that drives the hardest tradeoff: everything is sized for the open, and correctness cannot be traded for it.
• Exact books (to the cent and the share): Every fill moves cash and shares in one transaction, and every night our books are matched against the clearing firm’s.
• Available while the market is open (99.99 % from 9:30 to 16:00 ET): Planned work happens outside market hours. When something must give, order entry and cancels are the last to go.
• Auditability (every order event, write-once): Regulators ask years later what happened to an order and what the market was when it was routed, and expect an exact answer.

• Orders per second: ~210/s avg · 20 k/s at the open. 5 M orders over a 6.5-hour session (23,400 s) ≈ 214/s. The first minutes after 9:30 run at 20 k/s, about 100× the average. Size for the open, not the day.
• Quote messages without conflation: ~22 B/s. 3 M phones × 15 watched symbols = 45 M subscriptions. People watch the busiest symbols, ~500 updates/s each at the open: 45 M × 500 ≈ 22 B messages/s. No fleet can write that to sockets and no person can read it.
• Frames with a 250 ms tick: ~12 M frames/s. One frame per phone per tick: 3 M × 4/s = 12 M frames/s, carrying at most 45 M × 4 = 180 M symbol updates, ~125× fewer than forwarding everything. At ~20 B per symbol that is ~3.6 GB/s in all.
• Quote edge servers: 60 at peak · run 100. 3 M sockets ÷ 50 k per server = 60 servers, each pushing 50 k × 4 = 200 k frames/s, ~60 MB/s. Run 100 (30 k sockets each) so a zone loss or a reconnect storm fits.
• Latest-quote table: ~1 MB. 10,000 symbols × ~100 B (bid, ask, sizes, last, previous close, high, low, status, sequence) = 1 MB. At the open about half change in each 50 ms batch: ~500 KB × 20/s = 10 MB/s per subscriber. The whole market fits on every server.
• Ledger writes at the open: ~300 k rows/s. 20 k fills/s × ~10 rows (execution, order, entry, 4 postings, cash and position rows, outbox) = 200 k, plus 20 k orders/s × ~5 rows = 100 k. Across 32 shards: ~9 k rows and ~1.25 k transactions per shard per second, comfortable for Postgres.
• Portfolio reads at the open: ~50 k/s. 3 M people load their portfolio in about a minute: 3 M ÷ 60 s = 50 k reads/s of cash, positions and open orders. On the shards they would compete with the writes that move money, so they are served from a cache.
• Audit trail growth: ~7.5 TB/yr. 5 M orders × ~6 events (received, sent, acknowledged, fills, done) × ~1 KB = 30 GB/day; × 252 trading days ≈ 7.5 TB/yr, kept for years in write-once storage. Small: the hard part is that it is complete.

• Trading app (iOS · Android · web): Holds one WebSocket for prices and its own order updates, and calls REST for orders and the portfolio. It computes the portfolio value itself, cash plus each position times the latest streamed price. It creates a client order id when an order ticket opens and reuses it on every retry of that order.
• API gateway (auth · admission control): Terminates REST, authenticates the session and runs admission control: 5 orders a second per account and an order-entry token bucket sized to what the order path sustains, 25 k/s. Excess orders wait in a queue for up to 2 s, then get 503 with a jittered Retry-After. Cancels bypass the bucket, because they reduce risk and load.
• Order service (checks · state machine): Owns orders. Runs the pre-trade checks (market session, halts, price collar, buying power, shares owned) and persists the order with its hold in one transaction on the account’s shard before anything is sent. Applies every execution report in one transaction, idempotent on the execution id. Also owns alerts and broker-held stops, so everything that can create an order goes through it.
• Account DB (Postgres · 32 shards by account): Everything about one account on one shard: orders and their events, executions, a double-entry journal in which cash and each stock are instruments, cached cash and position rows with their holds, alerts and held stops. Sharding by account makes every order and every fill a single-shard transaction. House accounts are append-only with no cached row, so nothing contends on them.
• FIX gateway (sessions · sequence numbers): The only component that speaks FIX, holding several sessions to each venue. Stores every outbound message with its sequence number before writing it to the socket, and every inbound one before passing it on, in a log replicated to a standby, so a reconnect is repaired with resend requests. Refuses to send a ClOrdID twice, and never turns an unanswered order into a new one.
• Feed handler (A/B lines · 1 MB table): Consumes the consolidated feed once, on two redundant lines, using whichever copy of each sequence number arrives first. Keeps the latest quote, last sale and trading status for 10,000 symbols in about 1 MB. Every 50 ms it publishes one batch of the symbols that changed, with each interval’s high and low, to the quote edges, the order service and the trigger engine.
• Quote edge (WebSocket · 100 servers): WebSocket servers, sized for 50 k sockets and run at ~30 k. Each holds the whole latest-quote table from the batches, and per connection a subscription set and a dirty set; every 250 ms it sends one frame with the newest values of the subscribed symbols that changed, and skips the tick for a socket still draining its last frame. It also delivers the account’s own order updates.
• Portfolio cache (Redis · per account): Cash, buying power, positions and open orders per account, kept current from order and fill events on the bus and versioned by the account’s event sequence. Serves the ~50 k portfolio reads a second at the open, so the shards see only orders and fills. It is also what the app shows, read-only, when the order path has to be protected.
• Trigger engine (sorted thresholds by symbol): Watches the price for what the venues do not hold: stops, price alerts and good-til-cancelled orders overnight. Partitioned by symbol; per symbol, below thresholds sorted high to low and above thresholds sorted low to high, compared with each batch’s low and high, so an update costs two comparisons unless something fires. Rebuilt from a compacted topic on the bus.
• Event bus (Kafka · from the outbox): Fed by the transactional outbox on every shard and keyed by account, so one account’s events stay in order. Consumers: the portfolio cache, the quote edges (each event to the edge holding that account’s socket), notifications, the trigger engine and the audit archive. At least once, so every consumer dedupes on the event id.
• Venues (exchanges · market makers): Exchanges and wholesale market makers that execute our orders and return execution reports. Each FIX session has its own sequence numbers and a message-rate throttle. Each venue also sends a drop copy, a second session carrying a copy of every execution, which is compared with our books every few minutes.
• Consolidated feed (SIP or vendor · 1 M updates/s): The US consolidated quote and trade feeds (the SIPs) or a vendor’s normalised version: the national best bid and offer, last sales and halts for every listed stock, around 1 M updates a second at the open. Delivered on two lines, so a packet lost on one is usually present on the other.
• Audit archive (write-once object storage): Every order event (received, rejected, sent, acknowledged, filled, cancel requested, cancelled) with the time, the device, the route and the quote when it was routed. Written from the bus into object storage under a compliance lock, partitioned by day and shard. The source of the next morning’s Consolidated Audit Trail report and of any best-execution review.
• EOD reconciliation (nightly, on replicas): After the close, joins the clearing firm’s trades, positions and cash with our ledger: trades by execution id, positions by account and symbol, cash by account. Also checks our own invariants: every hold has an open order, every cached row equals its postings. Breaks become cases with both sides of the evidence; it never corrects money by itself.
• Clearing firm (settles T+1): The clearing firm (or our own clearing entity) that settles trades through the clearing house on T+1 and holds the firm’s positions at the depository. Its end-of-day files are the outside view of what every account owns and owes, so any disagreement with them is a break to explain.

1. Market buy, from tap to fill: The core write path. The order is checked and recorded with its cash set aside before it leaves, and the fill moves cash and shares in one transaction, so a crash at any step leaves a state that can be resumed.
   1. The app posts a market buy for 10 AAPL with the client order id it created when the order ticket opened.
   2. The order service checks the market session, the halt status and the price collar against its in-memory copy of the latest quotes.
   3. In one transaction on the account’s shard, it inserts the order as new, places the hold and writes the order event to the outbox.
   4. It hands the order to the FIX gateway, which numbers it, stores it and sends a NewOrderSingle to the chosen venue.
   5. The venue acknowledges and fills it; the gateway journals each execution report and passes it to the order service.
   6. The order service books the fill in one transaction: the execution, the order’s state, the ledger entry, the cash and position rows, and the hold.
   7. The outbox event reaches the bus; the edge holding the user’s socket pushes the fill, the portfolio cache updates and the audit archive appends it.
   8. The app adds 10 AAPL to its positions and values them from the next price tick.
2. Live prices to 3 M phones: The main read path and the biggest stream in the system. One feed is consumed once and conflated twice, so the cost grows with phones times ticks, not with how often prices change.
   1. The feed handler takes the consolidated feed on two lines and keeps the newest quote for each symbol.
   2. Every 50 ms it publishes one batch of the symbols that changed to every quote edge.
   3. The app connects, subscribes to its watchlist and positions, and gets a snapshot from the edge’s own table.
   4. The app loads cash, positions and open orders from the portfolio cache.
   5. Every 250 ms the edge sends one frame with only the subscribed symbols that changed since the last frame.
   6. A phone on a weak connection falls behind, and the edge skips its ticks instead of queueing them.
   7. On every frame, the app recomputes the portfolio value as cash plus each position times its latest price.
3. 9:30: the open, and a gap that fires stops: The scale-breaking case: order flow jumps ~100× in seconds, 3 M apps connect, and a stock that opens 8 % lower crosses 30 k stops at once. Capacity is in place before the surge, and admission decides what waits.
   1. At 9:00 a scheduled scale-up brings quote edges, gateways and order services to the forecast peak, and every FIX session is already logged on.
   2. 3 M apps connect within minutes; each edge admits new sockets at a capped rate and the apps back off with jitter.
   3. Portfolio screens load from the portfolio cache, so the shards see only orders and fills.
   4. 20 k orders a second arrive; the gateway admits them through the order-entry bucket and holds the excess in a short queue.
   5. TSLA opens 8 % lower; the first batch’s low crosses 30 k stop prices and 120 k price alerts at once.
   6. Each triggered stop enters the order service as an order whose client order id is derived from the stop, in its own lane.
   7. Each fired alert is marked fired exactly once and handed to notifications in a paced lane.
   8. If shard commit latency passes its threshold, the gateway sheds charts and news, then serves portfolios read-only while order entry keeps its capacity.
4. The FIX session drops with orders in flight: The failure that separates a broker from a toy. An order whose acknowledgement never arrived has an unknown outcome, and it is resolved by repairing the session and asking, never by sending it again.
   1. Orders 7781 and 7790 go out as sequence numbers 50,112 and 50,113, then the connection to the venue drops before any reply.
   2. The gateway reconnects and logs on; the venue’s sequence number is ahead of what we received, so the gateway asks for the gap.
   3. The venue resends what we missed, including the acknowledgement and fill for 7781, which are booked exactly once.
   4. Order 7790 still has no reply once the session is level, so the gateway asks the venue for its status.
   5. The venue reports the order as unknown, so 7790 is marked rejected, its hold released and the customer asked to place it again.
   6. If the gateway process itself dies, a standby takes over the session from the replicated message store with the same sequence numbers.
5. The close and the nightly reconciliation: The background path that proves the day. Day orders expire, the audit trail becomes the regulator’s report, and every trade, position and dollar is matched against the clearing firm’s records before the next open.
   1. At 16:00 the venues expire unfilled day orders, and each expiry releases its hold.
   2. The day’s order events, already in the audit archive, become the regulator’s report before the next morning.
   3. The reconciliation job pulls the clearing firm’s trade, position and cash files for the day.
   4. It joins trades by execution id, positions by account and symbol, and cash by account against the ledger.
   5. It checks our own invariants: every hold belongs to an open order, every cached row equals its postings, every order’s filled quantity equals its executions.
   6. A break: the clearing firm has a 50-share fill we never booked, so a case opens and the trade is booked as of its trade date.
   7. Before the next session, GTC orders are queued to be sent again as day orders and the trigger engine reloads its thresholds.

Ask: The feed carries a million updates a second and 3 M phones want prices. Why not forward every update to everyone who subscribed?

Good answers name: Consume once, conflate twice (50 ms batches to every edge, a 250 ms tick per socket), portfolio computed on the phone, Forward every update through a topic per symbol, Server computes each user’s portfolio value and pushes it, Clients poll a quotes endpoint every second.

Our pick: The feed handler consumes both lines of the consolidated feed, keeps the latest quote, last sale and status for 10,000 symbols in a 1 MB table, and every 50 ms broadcasts one batch of the changed symbols, with each interval’s high and low, to every quote edge: 100 servers, sized for 50 k sockets and run at ~30 k. Each socket has a subscription set (at most 50 symbols) and a dirty set; on a 250 ms timer the edge writes one binary frame with the newest values of the dirty symbols and clears the set, and skips the tick if the socket still holds its last frame. A new or reconnected socket gets a snapshot from the edge’s table, and each value carries the feed’s per-symbol sequence, so the phone never moves a price backwards after switching edges. The phone multiplies its positions by those prices for the portfolio value; the server only tells it when positions change. That is 12 M frames and at most 180 M symbol updates a second at the peak, against ~22 B messages without conflation.

1. An edge server dies with 30 k sockets on it. What happens?
   The phones see the socket close and reconnect after a jittered delay of a few seconds, through the load balancer, to the other 99 edges, about 300 each. Each gets a snapshot from its new edge’s table and carries on; there is nothing to replay, because quotes supersede themselves and order updates resume from the account sequence the phone already has. Admission per edge keeps the TLS handshakes from piling up, and because we run 100 servers for a 60-server load, losing one, or a zone, fits.
2. Why does every edge take the whole market instead of only the symbols its clients watch?
   Because the whole market is 1 MB and ~10 MB/s at the open, which is nothing for a server, while routing by subscription needs a registry of which edge wants which symbol, updated as millions of people scroll. With 30 k phones per edge, nearly every popular symbol is wanted on every edge anyway. Topic routing becomes the right answer when the instrument count explodes, as with options, which have millions of contracts.
3. TSLA updates 5,000 times a second at the open. What does a user actually see?
   Four prices a second, each the newest at the moment its frame was built. The batch carries the interval’s high and low as well, so the day’s range is right and the triggers do not miss a spike between frames. Charts are built from every trade by a stream job producing one-minute candles, not from the conflated stream, so the chart is exact even though the ticker is sampled.
4. How would you know the prices on phones are fresh?
   Two measurements. At the edge, the gap between the feed’s timestamp on each value and the moment it is written to a socket, as a p99 per edge. On the phone, a sampled report of the age of the newest price it drew. The target is p99 under 500 ms with an alert on either side, plus a check that flags any active symbol whose value has not changed for 30 s during market hours, which usually means a feed line is down.
5. What changes at ten times the users?
   30 M connected phones is 120 M frames a second and about 600 edges at 50 k sockets each; the edge tier scales linearly because it holds no per-user state beyond subscriptions. The feed side barely changes: one feed handler pair and 1 MB of state, broadcast through a tier of relays, because 600 subscribers at 10 MB/s is too much for one sender. The tick could stretch to 500 ms on weak networks as a pressure valve, which halves the cost and is hard to notice.

Ask: The user taps Buy, the network drops, the app retries. How do you guarantee exactly one order, and that the cash is actually there?

Good answers name: Client order id as a unique key; checks, hold and order row in one shard transaction before sending, An idempotency key in Redis in front of the order service, Read buying power, send, and debit when the fill arrives, Send to the venue first, record the order when it is acknowledged.

Our pick: The app creates a client order id when the order ticket opens and sends it with every attempt. The order service runs the cheap checks from memory (market session, halt status, symbol tradable, a price collar, a confirmation for a limit far from the market) and then one transaction on the account’s shard: insert the order as new under a unique key on (account_id, client_order_id); place the hold with a conditional update that requires cash minus held to cover qty × limit, or qty × ask × 1.05 for a market buy, which is sent as a limit at that collar; for a sell, hold the shares the same way; and write the order event to the outbox. A duplicate returns the existing order, and a duplicate with a different body is a 422. Only after the commit does the order go to the FIX gateway, which refuses a ClOrdID it has already sent, and a sweeper resolves any order left in new for 2 s by asking the gateway, so the crash between commit and send is harmless. Holds are released in proportion to each fill and in full on a cancel, rejection or expiry.

1. The order service crashes after the commit but before it calls the FIX gateway.
   The order sits in new with its hold. A sweeper finds orders in new for more than 2 s and calls send again; the gateway checks its store for that ClOrdID and sends it if it has never seen it, or returns the existing state if it has. The customer saw "order placed" after the commit, which was true: the order was recorded and its money set aside, and it reaches the venue a couple of seconds late. Time in new is alerted on, because it should be near zero.
2. Why hold the cash when the order is placed instead of debiting it on the fill?
   Because between placing and filling, the cash is promised. Without the hold, a resting limit buy for $1,500 and a market buy for $1,500 against $2,000 both pass, both can fill, and the account is $1,000 short at settlement, a loss the broker carries. The hold makes the check and the reservation one conditional update on one row, so concurrent orders serialise. It is the pending-entry idea from payments: available drops now, cash moves on the fill.
3. A market buy for 10 shares with the ask at $189.50, and the price runs before it fills. How much do you hold?
   The order is sent as a limit at the ask plus 5 %, $198.98, so the hold is 10 × $198.98 and no fill can cost more than the hold. If the stock runs past the collar, the order rests unfilled instead of buying at any price, and the customer can cancel or wait; after a halt reopens far away, that protection is worth more than the convenience. Whatever the fill does not use is released with it.
4. Why not let the venue reject an order the customer cannot pay for?
   The venue does not know our customers; it knows us. Every order we send is ours, and if the customer cannot pay at settlement, we pay. So the check is the broker’s job, and it has to happen before the order leaves, because after that the only remedy is a cancel that may arrive after the fill.
5. A regulator asks what happened to one order last March. What can you show?
   Every event of that order, in order, from the audit archive: received, with the device and session; the checks it passed; the quote when it was routed and where it went; the venue’s acknowledgement and each fill; any cancel. The events are written in the same transaction as each state change and copied to write-once storage within seconds, so the record cannot disagree with the books and cannot be edited afterwards. The same records feed the Consolidated Audit Trail report each morning.

Ask: You sent an order to a market maker and the connection dropped before any reply. Did it arrive? What do you do?

Good answers name: Repair the session with sequence numbers and resend requests, then ask about anything still unanswered, Send it again as a new order after a timeout, Reset sequence numbers and start a fresh session on reconnect, Cancel-on-disconnect at the venue.

Our pick: The FIX gateway writes every outbound message to its store with its MsgSeqNum before writing it to the socket, and every inbound message before passing it on, in a log replicated to a standby. On reconnect it logs on as the same session with its next numbers; a gap in what the venue sent is requested with a ResendRequest and replayed with PossDupFlag, and executions are deduplicated on (venue, ExecID). Messages the venue missed from us are resent from the store with the same ClOrdID. Any order still without a reply 5 s after the session is level gets an OrderStatusRequest, or a cancel naming its ClOrdID where status requests are not supported; an answer of unknown order marks it rejected and releases its hold, and the customer is asked to place it again. Nothing is ever re-sent under a new ClOrdID on our own initiative. Each venue’s drop copy, a second session with a copy of every execution, is compared with our books every few minutes, so a fill the main session never delivered is found within minutes rather than at night.

1. The status request times out too, and the customer is staring at "pending". Why not just resend?
   Because the costs are not symmetric. Waiting costs the customer a minute of uncertainty; a wrong resend costs a duplicate fill, possibly in a moving market, that someone has to unwind. So the order stays sent and the app says so honestly, the gateway keeps asking with backoff, and after a minute it tries a cancel naming the ClOrdID, whose answer (cancelled, too late to cancel, or unknown) settles the question. Past five minutes it pages a person.
2. The venue said unknown order, you marked it rejected, and two minutes later a fill for it arrives.
   Book the fill: an execution report is evidence of a trade, and the trade will settle whatever we think. The state machine lets a rejected, not-received order become filled only through this path, and doing so opens a case, because it means the venue’s answer was wrong. The customer now owns shares they were told they did not buy, which support handles; it is exactly why we never resend on their behalf, since that would have been a second fill.
3. How many sessions, and what limits them?
   Venues throttle each session, often to a few hundred or a thousand messages a second, and orders, cancels and replaces all count. At 20 k orders a second plus cancels across five venues, that is around 40 sessions, eight per venue, with orders spread across them by account so one session’s trouble touches a fraction of customers. The router steers around a session near its throttle or answering slowly, which is also how a venue having a bad morning is avoided.
4. The gateway process crashes. Can another one take over the session?
   Yes, because the store is replicated. The standby takes a lease on the session before it logs on and the old process is fenced out, since a venue allows one logon per session identity and two processes interleaving sequence numbers would corrupt it. It logs on with the stored numbers, and the venue sees a short disconnect, not a new session that has lost its history.
5. How do you test this when the network in staging never drops?
   With a FIX venue simulator that misbehaves on purpose: it drops the connection after a chosen message, delays acknowledgements, sends duplicates and gaps, answers status requests with unknown order and sends late fills. Every transition of the order state machine has a test driven through it, and in staging the simulator injects these faults at a low rate all day, so the recovery paths run constantly rather than only during an incident.

Ask: A fill arrives. How do cash and shares change together, and how do you prove at night that your books match the street’s?

Good answers name: One transaction per fill on the account’s shard: execution, order, a double-entry entry with cash and shares as instruments, cached rows, A cash service and a position service, each updated from fill events, Treat the clearing firm’s files as the books and keep no ledger of our own, Coalesce one order’s fills into a single entry every 100 ms.

Our pick: The ledger is double entry with cash and each stock as instruments, and every entry balances per instrument. A buy fill of 10 AAPL at 189.52 is one entry: customer cash −$1,895.20 and house street cash +$1,895.20, customer AAPL +10 and house street AAPL −10. The same transaction inserts the execution under a unique (venue, ExecID), moves the order to partially filled or filled, updates the cached cash and position rows, releases the matching part of the hold and writes the outbox row: about ten rows and one commit, ~1.25 k transactions a second per shard at the peak across 32 shards. House accounts have no cached row; their postings are append-only and summed per shard every second, as in Design Venmo, so they never become a hot row. Sale proceeds can buy at once but cannot be withdrawn until they settle on T+1. Every night the reconciliation job matches trades by execution id, positions by account and symbol and cash by account against the clearing firm’s files, checks that customer plus house positions equal the firm’s position at the depository, and turns each break into a case with both sides of the evidence; it never corrects money by itself.

1. The fill arrives while the account’s shard is failing over.
   The FIX gateway has the report in its journal and keeps retrying the order service, which cannot commit until the new primary is up, typically within 30 s. Nothing is lost and nothing is applied twice, because the unique key on the execution makes a retry after an ambiguous timeout harmless. The customer sees the fill late, and the venue never waits on us: the session keeps reading whatever the database is doing.
2. An order for 100 shares fills as 30, then 50, then 20.
   Three executions, three entries, each moving its own cash and shares and releasing its share of the hold. The order goes from sent to partially filled twice and then to filled, with our cumulative quantity checked against the venue’s CumQty on every report. The average price is computed from the executions, never kept as a running float. When the last 20 fill, the rest of the hold, the gap between the collar and the actual prices, is released.
3. The clearing file shows a trade you never booked.
   Theirs is what settles, so the trade is real until shown otherwise. An operator books the execution through the order service as of its trade date, which posts the entry, updates the position, notifies the customer and records the correction in the audit trail. Then we find out why: usually an execution the session lost, which the intraday drop-copy comparison should have caught within minutes, so it is also a bug.
4. A stock splits 4-for-1 overnight. What happens to the books?
   A corporate action job runs before the open: for each holder, an entry that quadruples the shares and leaves the total cost basis unchanged, a quarter per share, and the previous close in the quote table is adjusted so the app does not show a 75 % loss. Open orders in the symbol are cancelled or re-priced by policy, and customers are told. The clearing firm’s position file the next night confirms the new quantities.
5. Why not one database for every account?
   At 40 k transactions a second at the peak, one primary would be the bottleneck and the single thing every customer depends on at 9:30. Sharding by account costs nothing in consistency here, because no order or fill crosses accounts; only the house accounts do, and they are append-only. 32 shards keep each at ~1.25 k transactions a second with room for twice that, and a bad shard touches 1/32 of customers.

Ask: Who watches the price for a limit order, a stop order and a price alert, and why not handle them all the same way?

Good answers name: Rest limit orders at the venue; hold stops, overnight GTC orders and alerts in a trigger engine of per-symbol sorted thresholds, Hold every limit order at the broker and send it when the quote crosses, Store alerts in a table and query for crossed thresholds on every price change, Evaluate alerts on the phone.

Our pick: Non-marketable limit orders go to the venue and rest there as day orders. The trigger engine is partitioned by symbol across a handful of servers, each with a standby; for each symbol it keeps below thresholds (sell stops, falling alerts) sorted high to low and above thresholds (buy stops, rising alerts) sorted low to high: ~10 M alerts and ~2 M stops in ~400 MB. Each 50 ms batch’s low and high, taken from last sales, are compared with the head of each list, so a quiet symbol costs two comparisons and a gap pops exactly the thresholds it crossed. A fired stop becomes an order whose client order id is derived from the stop id, so a double fire is one order; its shares were held when the stop was placed, and fired stops get a reserved fifth of order-entry capacity. A fired alert is marked fired by a conditional update and pushed through a paced lane. GTC orders are re-sent each morning as day orders. The engine rebuilds from a compacted topic of current trigger states, a few seconds of reading.

1. The trigger engine for TSLA restarts at 9:30:05, in the middle of the gap.
   Its standby has the same state, read from the same compacted topic and the same batches, and takes over within a second. If both are gone, a replacement reads the topic in a few seconds and compares the current batch with every threshold, so anything crossed in the meantime fires late but fires. Firing twice is harmless: a stop’s order id is derived from the stop, and an alert is marked fired by a conditional update.
2. 30 k stops fire on one batch. In what order do they go out?
   Highest stop price first for sell stops, the order in which the falling price crossed them, then by placement time within a price. They take a reserved fifth of order-entry capacity, about 5 k a second, so all of them are sent within six or seven seconds while customers place 20 k orders a second, and none waits behind a new order. Each is a collared market sell, so a stop in a collapsing price cannot sell for nothing.
3. Why trigger stops on last sales and not on quotes, which update more often?
   Because a stop means "if it trades at this price, get me out", and a quote can flicker through a level without any trade there, especially in a thin pre-open book. Triggering on last sales avoids firing on a quote spike nobody traded at. For alerts it is a product choice; we use the last price for both so users see one rule.
4. A user moves their stop from $230 to $225 while the price is at $226.
   The change is a conditional update in the account shard from the old price, published to the triggers topic, and the engine swaps the threshold. If the price crossed $230 before the change reached the engine, the stop has already fired and the change fails with already triggered, which the app shows. There is no window in which it fires at both prices, because each fire is keyed by the stop id.
5. How do you know alerts are not being missed?
   Replay. A daily job runs the day’s recorded batches through a fresh engine built from the morning’s snapshot and compares what should have fired with what did. Live, we track fires per batch per symbol and the lag from batch to push. A symbol whose thresholds were crossed but which fired nothing shows up the same day, not when a user complains.

Ask: At 9:30 order flow jumps a hundredfold in seconds and 3 M people open the app. What breaks first, and how do you keep order entry working?

Good answers name: Pre-scale on a schedule, admit orders through a sized bucket with fair shares, keep reads off the ledger, degrade in a fixed order, Reactive autoscaling, Reject over capacity with 429 and let clients retry, Accept every order into a queue and process it asynchronously.

Our pick: Capacity follows a calendar: by 9:00 the quote edges, gateways and order services are at the busiest of the last 20 opens times 1.5, more on earnings, index rebalance and macro data days, while the shards and FIX sessions are sized for the open all day. Orders placed overnight are a known quantity and are sent to their venues before the open, so they are executed at the opening rather than added to the 9:30 spike. At the gateway, an order-entry bucket of 25 k/s (the rate measured in load tests), 5 orders a second per account, a reserved fifth for triggered stops, and a 2 s queue before a 503 with a jittered Retry-After; the app retries with the same client order id, and cancels bypass the bucket. Reads never touch the ledger primaries: the portfolio comes from the portfolio cache and history from replicas. A degradation ladder, flipped automatically on shard p99 commit latency over 50 ms or by hand, sheds charts, news and search first, then serves portfolios read-only from the cache, and touches order entry last. There are no deploys during market hours, and each edge admits 2,000 new sockets a second against jittered reconnects.

1. Why not just autoscale?
   Because autoscaling answers after the fact, and at 9:30 the answer arrives minutes late: by the time new instances pass health checks, the surge has peaked. The open happens at a known second every day, so a schedule is strictly better for it, and autoscaling stays as a second layer for the unscheduled spike, like a halt lifting at 2 pm. Databases and FIX sessions cannot be scaled on the fly anyway; they are sized for the open and run at that size all day.
2. The bucket is full and a customer is trying to sell into a falling market. Do they wait?
   A new sell waits like any order, at most 2 s, because the bucket protects the shards the sell needs too. The customer’s cancels never wait, and their stops have a reserved lane. Fairness across customers comes from the per-account share: a script sending 100 orders a second gets 5, so one heavy user cannot push a crowd of single orders into the queue.
3. What about the orders people placed overnight?
   They are known by 9:00, a few hundred thousand of them. They are sent to their venues before the open, paced over the pre-open, so the venue has them when trading starts. Where a venue cannot take them early, they are released at 9:30 in placement order at a reserved share of each session’s throttle, spread over tens of seconds rather than the first one.
4. How do you know the bucket size is right?
   A weekly load test, and one after every change to the order path, replays a real open at 1.5 times its rate against a production-shaped stack and finds the rate at which shard p99 commit latency crosses 50 ms; the bucket is set below it. In production the admission queue’s depth and wait time are dashboards with alerts: a bucket that never fills protects nothing, and one that fills on ordinary opens is too small.
5. What does read-only mode look like to a customer?
   The portfolio, positions and prices still show and still move, because prices come from the quote edges and positions from the cache, with a line saying balances are as of a few seconds ago. What stops is everything else that writes to the account: transfers, settings, new alerts. The customer can see what they own and can still trade it, which is all that matters at a bad open.