Design a Multiplayer Game Backend

• Log in and load the lobby: Account, profile, locker and friends in a couple of seconds. When demand is more than the backend can take, a waiting room with a real position and estimate, never a wall of errors.
• Friends, presence and parties: Parties of up to four. The party leader queues; matchmaking is an existing service that takes the party and hands back a formed match of 100 players and a region.
• Start each match on a dedicated server within seconds: From match formed to every client connected in under 10 s: a server process in the chosen region, on the players’ build, already running.
• Real-time movement, building and shooting for 100 players: Your own movement responds on the next frame, other players move smoothly, and the server decides every hit, judged as the shooter saw it.
• Matches of up to 25 minutes: The map shrinks until one player or squad is left. Eliminated players leave early and can queue again at once, so a match empties as it goes.
• Rewards, stats and inventory after each match: XP, battle pass progress, quests, lifetime stats and item grants, applied exactly once per player per match and visible in the lobby within seconds of being eliminated.
• Store and wallet: Buy currency with real money through the platform stores and spend it on items. A purchase debits and grants atomically, exactly once, and a refund reverses it.
• Cheat resistance: The server rejects impossible inputs and never sends a player what they should not know. Client anti-cheat and statistical detection feed bans.
• Out of scope: Matchmaking internals (see Design a Matchmaking System), leaderboards (see Design a Real-Time Leaderboard), voice chat, rendering and game design, patch downloads to clients, and the internals of the anti-cheat driver.

• Scale (3 M concurrent players · 30 k matches): Peaks follow the evening in each region. A season start or a big launch brings 2–3× the usual peak inside an hour, most of it in the first ten minutes.
• Responsiveness (own input next frame · RTT < 80 ms): Prediction hides the round trip for your own movement. Matchmaking keeps every player under ~80 ms to the region, and the 30 Hz tick adds up to 33 ms before an input is simulated.
• Fair hit detection (server decides · rewind ≤ 200 ms): This is the requirement that drives the hardest trade-off: players disagree about where everyone was, and one view has to win. Favouring the shooter, within a cap, is the choice.
• Bandwidth per player (~15 KB/s down · ~2.4 KB/s up): Has to fit poor home and mobile connections, and sets the egress bill: ~54 MB per player-hour and ~360 Gbps across the fleet at peak.
• Match start (allocation p99 < 2 s · all connected < 10 s): Players will not wait minutes for a machine to boot. Capacity must already be running in every region: allocation picks a warm server, it never creates one.
• Match reliability (< 1 in 1 000 matches lost): A crashed match is lost, not failed over. State changes 30 times a second, and replicating it costs more than the rare loss, so crashes are kept rare and compensated.
• Correct progression and money (rewards exactly once · purchases atomic): A result applies once per (match, player) however often it is delivered, and a purchase debits and grants in one transaction. A retry never mints currency.
• Live services (99.95 % · degrade, don’t fall over): Under a surge, logins wait at a fixed admission rate rather than letting retries take down the profile database. A match in progress never depends on the backend.

• Concurrent matches: ~30 k. 3 M concurrent ÷ 100 per match = 30 k. Two effects roughly cancel: about 10 % of players are in the lobby or the queue at any moment, while eliminated players leave early, so a running match averages fewer than 100 people. Plan on 30 k.
• Match starts per second: ~20/s · ~50/s at a season start. 30 k concurrent ÷ 1 500 s (25 minutes) = 20 matches/s, so 2 k players/s are placed on servers. Each of them later produces exactly one result, so the post-match pipeline writes ~2 k results/s. A season start reaches ~50 starts/s, paced by the login queue.
• Downstream per player: ~15 KB/s (~120 kbps). Interest management keeps a snapshot to the ~40 entities near you, and delta compression against the last snapshot you acknowledged leaves ~12 B per moving entity. ~40 × 12 B plus headers and events ≈ 500 B, × 30 Hz = 15 KB/s. Without either, the full state of 100 players is ~4 KB a tick (120 KB/s) and does not fit in one packet.
• Upstream per player: ~2.4 KB/s. An input packet is ~50 B of payload (sequence, acks, buttons, view angles, and the previous two inputs again in case of loss) plus 28 B of UDP/IP headers ≈ 80 B, × 30 a second = 2.4 KB/s. Clients send intent, never positions, so upstream stays tiny.
• Game server hosts at peak: ~1 500 (20 matches each). A match’s game loop is mostly single-threaded and needs about one core at its busiest (the landing, the final circles) plus ~2 GB of memory. A 32-core host runs ~20 matches and keeps the rest for networking and spikes: 30 k ÷ 20 = 1 500 hosts, plus the warm buffer.
• Egress per host: ~240 Mbps · ~360 Gbps fleet-wide. 100 players × 15 KB/s = 1.5 MB/s = 12 Mbps per match; × 20 matches = 240 Mbps per host, a few percent of a 10 Gbps NIC. Fleet-wide, 3 M × 120 kbps ≈ 360 Gbps. Packets bind before bits do: 20 × 100 × 30 = 60 k packets/s out of each host, and as many in.
• Warm buffer of ready servers: ~3.6 k (~12 % of the fleet). Bringing a new host into service takes ~3 minutes, so ready processes must cover the starts in the meantime: 20 starts/s × 180 s ≈ 3 600 servers across ~10 regions, ~12 % on top of 30 k. A season start is covered by raising capacity ahead of time, not by this buffer alone.
• Logins at a season start: ~5 k/s · ~50 k/s with retries. 3 M players opening the game within ~10 minutes = 5 k/s, against ~550/s on a normal evening (3 M sessions of ~90 minutes). Clients retrying every few seconds turn that into ~50 k/s of attempts. Each login reads a ~50 KB profile, so 5 k/s is ~250 MB/s from the player store. This, not the game servers, is what falls over first.

• Login queue (waiting room): The front door during surges. Admits logins at a rate the backend has shown it can take and gives everyone else a signed ticket with a position and an estimated wait. The admission rate is a control loop on backend health, so neither demand nor retries can push load past it.
• Waiting-room state (Redis · two counters): Per region, the next ticket number (INCR on join) and the admitted watermark the admission controller advances by R each second. A position is ticket minus watermark, so the queue costs O(1) per player whatever its length. Tickets are signed and carry their number, so losing this store loses no one’s place.
• Platform gateway (sessions · WebSocket): Trades an admission token and a platform credential for a session, authenticates every lobby call and routes it to the live services. Holds one WebSocket per online player for party invites, presence and match.ready pushes. Nothing on a match’s real-time path goes through it.
• Store & wallet (transactional · ledger): Sells items for currency and currency for real money. Spending is one transaction on the player’s shard: insert the client’s purchase id (unique), append a debit to the wallet ledger if the balance covers it, insert the entitlement. Real-money credits are unique on the platform’s transaction id, and refunds post reversing entries.
• Payment platforms (console stores · PSP): PlayStation, Xbox, Nintendo, Apple, Google, Steam, and a card provider for direct sales. Each returns a signed receipt or a server-side purchase record we verify before crediting currency, and each sends its own refund and chargeback notifications that we must consume.
• Game client (predicts · interpolates): Samples input every frame and sends it to the game server 30 times a second over UDP. Predicts its own movement at once and reconciles against the server’s answer, and draws everyone else ~100 ms in the past, interpolating between snapshots. Runs the client anti-cheat, whose session token is required to join a match.
• Friends & parties (presence · party state): Friends lists, presence and parties. Presence is pushed only on state changes (online, in a match, back in the lobby), ~4 k/s, while details such as players left are read when someone opens the list. The party leader’s queue request goes from here to matchmaking with the whole roster, and the party survives a lost match because it never lived on the game server.
• Matchmaking (existing service): The existing matchmaking service (see Design a Matchmaking System), not designed here. Takes a party’s ticket, forms a 100-player match in a region chosen by measured latency, asks the allocator for a server, and pushes match.ready with the address and join tokens through the gateway.
• Profile service (account · progression · items): Owns the account (credentials check, bans), progression (XP, level, battle pass tier, quests), lifetime stats and the item inventory. Read once at login for the lobby, written mostly by the post-match pipeline, one transaction per result. Never called by a game server during a match.
• Player data store (sharded by account id): Accounts, profiles, inventory, the wallet ledger and the applied-results table, all keyed by account id on the same shard, so a reward or a purchase is a single-shard transaction. A sharded relational store (MySQL with Vitess, or Postgres) gives the unique constraints idempotency needs. A profile is ~50 KB, read once per login.
• Game server (authoritative · 30 Hz · UDP): One process per match, holding the whole match in memory. Every 33 ms it applies the inputs that arrived, simulates movement, building and damage, and sends each player a delta-compressed snapshot of only what is relevant to them. Validates every input, lag-compensates hits by rewinding hitboxes, and writes nothing to the backend until a player’s match ends.
• Regional fleets (Agones on Kubernetes): A few clusters per region, ~20 game server processes per host, each a pod with its own UDP port. Keeps a buffer of Ready servers that the forecast raises before peaks, bursts into cloud capacity when owned hosts run out, and never removes an Allocated server, which is how a build update drains: old servers finish their match and exit.
• Server allocator (match registry): Turns a formed match into a running server in about a second. Picks a cluster in the requested region with Ready servers on the right build, allocates one through that cluster’s fleet, and records the match (id, roster, server, start time, state) with a signed join token per player. Watches server health so a crash becomes a match.aborted event, not silence.
• Post-match pipeline (idempotent on (match, player)): Consumes results, checks each against the match’s roster and the playlist’s caps, and applies XP, quest progress, stats and item grants through the profile service in one transaction guarded by a unique (match id, account id) row. Turns match.aborted into time-played compensation. Leaderboards and anti-cheat analytics read the same topic as separate consumers.
• Results queue (Kafka · key = account id): The durable buffer between game servers and the backend. match.results holds one record per (match, player), a result when that player’s match ends or a match.aborted from the allocator, keyed by account id so a player’s records apply in order; match.events holds match.started with the roster, keyed by match id. Retains days, so a profile store outage delays rewards but never loses them.

1. Match formed to first tick: Matchmaking hands over 100 players and a region; within seconds a warm server there is allocated, learns its roster, and every client is connected over UDP with a token. Nothing boots on this path.
   1. The party leader presses play; the friends service hands the whole party to matchmaking.
   2. Matchmaking forms a 100-player match in a region and asks the allocator for a server.
   3. The allocator picks a cluster in that region with Ready servers on the build and allocates one.
   4. The allocator records the match in its registry and signs a join token for each player.
   5. The game server learns it was allocated, reads its roster and opens a 60-second connect window.
   6. Matchmaking pushes match.ready to every player through the gateway socket.
   7. Each client connects over UDP with its token; the server checks the token and the anti-cheat session.
   8. When everyone is in or the window closes, the match starts and the allocator publishes match.started.
2. One tick: move, shoot, hit: The real-time core. Inputs go up 30 times a second, the server simulates, judges a shot as the shooter saw it, and sends each player only what is near them, encoded against what they last acknowledged.
   1. The client samples input, applies it locally at once, and sends it with a sequence number.
   2. Server tick: apply every input received since the last tick, after validating it, and simulate.
   3. A fire input arrives: the server rewinds the other players’ hitboxes to what the shooter saw and traces the shot.
   4. The server sends each client a snapshot of what is relevant to it, as a delta against its last acknowledged snapshot.
   5. The client reconciles: it resets its own player to the server state at last_input and replays the inputs after it.
   6. The client draws other players 100 ms in the past, interpolating between the snapshots around that time.
   7. The hit is confirmed in the next snapshot, and the victim’s client learns of it in the same tick.
3. Season launch: 3 M players at once: The scale-breaking case: demand arrives in minutes, not hours. Capacity is raised ahead from a forecast, logins wait in a queue at a rate the backend can take, and the first big purchase, the new battle pass, stays a single transaction.
   1. Hours before, the forecast raises each region’s fleet minimums and the new build is pre-pulled onto every host.
   2. Clients open the game; login is past its admission rate, so each gets a signed ticket and a position.
   3. An admission controller advances the watermark each second, and a ticket below it is exchanged for an admission token.
   4. The admitted client logs in at the gateway, which checks the account and issues a session.
   5. The lobby loads: profile, locker, friends and presence, in one aggregated call.
   6. The player buys the new battle pass with currency: one transaction debits the wallet and grants the pass.
   7. Another player tops up currency with real money; the store verifies the platform receipt and credits it once.
   8. Players queue, and the warm buffer absorbs the wave of allocations.
4. Game server crashes mid-match: The failure path. The match dies with its process, by design. What matters is noticing within seconds, getting players back to the lobby with their party, and compensating exactly once.
   1. A game server process crashes 14 minutes in, with 41 players still alive.
   2. Clients stop receiving snapshots and, after a few seconds of retrying, show “connection lost”.
   3. The allocator sees the server go Unhealthy, marks the match aborted, and publishes match.aborted for each player still in it.
   4. Clients ask the gateway, learn the match is gone, and return to the lobby with their party intact.
   5. The post-match pipeline turns each aborted record into compensation: XP for minutes played, no stats, no placement.
   6. The fleet replaces the capacity, and the crash is counted against the build’s crash budget.
5. Match ends: rewards written once: The asynchronous path. Each player’s result leaves the game server once, when their match ends, travels through a durable queue, and is applied exactly once to progression, stats and inventory, even when everything in between retries.
   1. A player is eliminated in 37th place; the server finalises their result from counters it kept in memory.
   2. The server publishes a signed result keyed by account id and keeps it until the broker acknowledges.
   3. The pipeline checks the result: the player was on this match’s roster, and the numbers are within the playlist’s caps.
   4. The profile service applies the result in one transaction, guarded by a unique (match id, account id) row.
   5. The pipeline commits its offset only after the transaction commits.
   6. The player, already back in the lobby, sees their XP and battle pass progress within seconds.
   7. When the match ends, the server shuts down; the fleet replaces it with a fresh Ready process.

Ask: Why pay for 1 500 servers? Why not let a player’s machine host the match, or let each client report its own position?

Good answers name: Dedicated authoritative server per match; clients send only inputs, Client authority: clients send positions and hits, the server relays, Peer-to-peer with one player as host, Deterministic lockstep: every client simulates, only inputs are exchanged.

Our pick: Dedicated authoritative servers. Clients send inputs 30 times a second; the server simulates at 30 Hz and is the only source of truth for positions, health, inventory, building and hits. It validates every input (movement axes clamped, turn rate within human limits, fire rate and ammo per weapon, packets per second per connection) and sends each player only what is relevant to them. Prediction hides the round trip for your own movement and lag compensation makes aiming fair, so authority costs the player nothing visible. Client anti-cheat is the second layer, at the boundary: the connect handshake requires a valid anti-cheat session token, and its detections, together with statistical checks on match telemetry, feed bans. Tick rate is a cost dial: 30 Hz suits 100 players on a large map, while 5v5 tactical shooters spend more per match on 64 or 128 Hz (Riot runs VALORANT at 128) because they have 10 players, not 100.

1. Why 30 Hz and not 60?
   Cost against what the game needs. A 100-player match at 60 Hz roughly doubles CPU per match, so ~10 matches per host instead of 20 and ~3 000 hosts, and doubles snapshot bandwidth to ~30 KB/s per player. The gain is ~17 ms less average wait before an input is simulated. For a battle royale with long sightlines, 30 Hz with good interpolation is the sensible point; a 10-player tactical shooter, where duels are decided in milliseconds, is where 64 or 128 Hz pays for itself.
2. Couldn’t you trust the client for movement and only check it for plausibility?
   Some games do: accept the client’s position if it is within maximum speed of the last one. The trouble is that every check becomes a tolerance, and cheaters tune themselves to sit just inside it, moving 10 % faster everywhere forever. Inputs-only is simpler to secure because there is no tolerance to exploit: the server computes the position itself. The cost, a round trip, is already paid for by prediction.
3. What does the server still have to trust the client for?
   Where the player is looking, when they pressed the button, and what they were seeing (the view time used for lag compensation). Aim is the hole: an aimbot sends perfectly valid inputs with superhuman precision. So the server caps turn rate and bounds the claimed view time, and the rest is detection: client anti-cheat looking for injected code, and statistics on match telemetry such as reaction times and snap angles.
4. How does an authoritative server stop wallhacks?
   By not sending what the player cannot see. Interest management already limits snapshots to nearby entities; going further, the server checks line of sight and includes enemies only when they are potentially visible, slightly early so they do not pop in. Riot has written publicly about doing this in VALORANT. It costs server CPU per player per tick, so a 100-player game usually does it more coarsely than a 10-player one.
5. Can one match use more than one core?
   Partly. The loop that steps the simulation is hard to parallelise because everything interacts with everything, but serialising 100 snapshots, physics queries and network I/O can move to worker threads, and engines do some of this. The bigger lever is many matches per host: 20 single-threaded matches use 20 cores in parallel with no locks between them, which is why per-match CPU is budgeted in cores.

Ask: Why UDP instead of TCP or WebSockets, and how do you fit 100 players into 15 KB/s?

Good answers name: UDP: unreliable delta snapshots, a reliable event channel, interest management, TCP or WebSockets for everything, QUIC datagrams or WebTransport, Full snapshots every tick, no deltas.

Our pick: A custom protocol over UDP. Each packet carries its sequence number, the latest sequence received from the other side and a 32-bit field acknowledging the 32 before it, so both ends know exactly which packets arrived without any retransmission queue. Snapshots are unreliable: the server keeps the last 32 it sent each client (~1 s), encodes each new one as a delta against the newest that client acknowledged, and falls back to a full snapshot if nothing was acknowledged for a second. Events that must arrive go in a reliable channel, repeated in every packet until acknowledged. Interest management uses a ~100 m grid plus a priority accumulator to fill a ~1 200 B budget with the most relevant entities. Positions are quantised (1 cm on a 2.5 km map is 18 bits per axis), angles to 16 bits, and unchanged fields cost one bit. The result is ~500 B per snapshot: 15 KB/s down and 2.4 KB/s up per player.

1. A player is on 3 % packet loss. What do they experience?
   Very little for movement: a lost snapshot is replaced by the next one 33 ms later and the interpolation buffer covers the gap, and their inputs ride three times so the server rarely misses one. Deltas get a little bigger because the baseline is older, and reliable events arrive a packet later. Past ~10 %, or with bursts of loss, the buffer runs dry, other players stutter, and the client shows a connection warning.
2. At the start, all 100 players are on the bus within a few metres of each other. Does interest management still help?
   Not much, since everyone is relevant to everyone. The priority accumulator still bounds each packet at ~1 200 B: the nearest and most-changed entities go every tick and the rest every second or third tick. On the bus there is little to send anyway, because everyone moves together; the heavy moment is the landing, which is why per-match CPU and bandwidth are budgeted for the busiest minutes, not the average.
3. Why not just run snapshots through zlib?
   A general-purpose compressor on a 500 B packet saves little and costs CPU on 3 000 packets a second per match. The large wins are domain-specific: only what changed since the acknowledged baseline, quantised to the precision the game needs, only for relevant entities. A small static entropy coder trained on real traffic adds perhaps 20 to 30 % on top, which is worth having at 360 Gbps.
4. Some networks block UDP. What then?
   A fallback that tunnels the same packets over TCP or a WebSocket on port 443 through a relay, accepting the head-of-line stalls. It serves the few percent of players on restrictive networks, and the client tells them their connection is degraded. The game protocol does not change, only the transport underneath it.
5. Every player learns the server’s IP address. How do you stop one of them knocking it offline?
   Packets that do not authenticate with the session key are dropped before any game logic, so junk floods are cheap to discard, but a volumetric flood still fills the link. So game traffic enters through providers with DDoS scrubbing at the edge, ports are per match and short-lived, and some studios route players through their own relay network so the real server address is never exposed. Rate limits per source address protect the connect handshake.

Ask: Your ping is 60 ms and the server is authoritative. How does movement feel instant, and whose screen decides whether a shot hit?

Good answers name: Prediction, 100 ms interpolation, lag compensation capped at 200 ms, No lag compensation: shooters lead their targets, Client-side hit detection, checked by the server for plausibility, Extrapolate other players forward instead of interpolating.

Our pick: The client runs the same movement code as the server on its own inputs and shows the result at once, keeping a buffer of inputs the server has not acknowledged; each snapshot carries the last input processed, and the client resets to that state and replays the rest, blending small errors and snapping only above ~1 m. Other players are drawn 100 ms in the past, interpolating between the snapshots around that time, which survives a lost packet. Each fire input carries the tick the client was rendering; the server rewinds other players’ hitboxes to that time (now minus the shooter’s latency and interpolation delay, the method Valve documents for the Source engine), traces the shot, and restores them. Rewind is capped at 200 ms: players under it hit where they aimed, players over it must lead. The trade-off is stated plainly: favour the shooter within the cap, and accept that a victim occasionally dies just after reaching cover.

1. The victim says they were already behind a wall when they died. Who is right?
   Both, on their own screens. The shooter saw them in the open 100 to 150 ms earlier and the server judged the shot in that past, which is the deliberate choice. The cap limits how far back this can reach, and matchmaking keeping everyone under ~80 ms keeps it to a fraction of a second. A killcam showing the shooter’s view helps players accept it.
2. How does reconciliation avoid jitter when the server disagrees slightly?
   It does not snap for small errors. If the corrected position is within a few centimetres, the simulation uses it at once but the rendered position blends toward it over ~100 ms. Only above a threshold, say 1 m, where something real happened such as a wall built in your path, does it snap. Identical movement code and fixed timesteps on both sides keep disagreements rare in the first place.
3. What does lag compensation cost the server?
   Memory: ~1 s of hitbox poses for 100 players at 30 ticks, ~600 KB per match. CPU: per shot, rewind only the players near the line of fire, trace, restore. An automatic weapon fires ~10 times a second, so a busy fight is a few hundred rewinds a second, small next to the simulation itself. Slow projectiles are simulated forward on the server instead of being rewound.
4. Could a cheater lie about their view time to rewind further?
   They can claim anything, so the server bounds it: the claimed view time must fall within the measured round trip plus the interpolation delay and a small tolerance, and never past the 200 ms cap. A claim outside that is clamped and counted, and a client that keeps doing it is flagged. It is the general rule: anything a client reports is a hint, bounded by what the server measured itself.
5. How would you know the netcode is working?
   Per region and per build: server tick time p99 against the 33 ms budget, snapshot size, loss and round trip per player, prediction corrections per minute and their size, and hit-registration complaints per thousand matches. A jump in corrections after a patch usually means client and server movement code have drifted apart.

Ask: Matchmaking hands you 100 players. How do they get a server in the right region within two seconds, at 20 matches a second and 50 on launch day?

Good answers name: Warm Ready pools on Agones, an allocator across clusters, forecast scaling and cloud bursting, Start a server on demand for each match, One VM per match, Managed game server hosting.

Our pick: Each region runs a few Kubernetes clusters with Agones, the open-source game server orchestrator. A Fleet per (build, playlist) keeps a buffer of Ready servers: processes already started, map loaded, waiting. The allocator picks a cluster in the requested region by Ready count and health and sends a GameServerAllocation with label selectors; Agones flips one server to Allocated, and Allocated servers are never scaled down or replaced. Packed scheduling fills hosts before starting new ones so the cluster autoscaler can remove empty hosts at the end of a peak. Capacity comes in layers by lead time: Ready processes, ~12 % of the fleet, answer an allocation in a second and are sized to last the ~3 minutes it takes to add a host; spare host headroom lets the fleet refill that buffer in ~30 s a process; and a per-region forecast sets minimums an hour ahead and books burst capacity before a season start. Baseline load runs on reserved or bare-metal hosts, and peaks burst into on-demand cloud hosts in the same metro. Updates drain: the new build’s Fleet scales up, the old one’s Ready count goes to zero, and its Allocated servers finish their match and exit, so a version change takes about one match length.

1. Packed scheduling puts 20 matches on one host. Isn’t that a big blast radius?
   Yes, and it is a deliberate trade. A host failure ends 20 matches, 2 000 players, at once. Spreading would lose only one or two per host, but it leaves every host partly full, so the autoscaler can never remove any and the fleet costs more after every peak. Hosts fail far less often than processes crash, so pack, and spread each cluster across availability zones so a zone failure is a fraction of a region.
2. A region’s Ready buffer hits zero during a launch. What happens?
   The allocator tries the region’s other clusters first. If all are empty it answers “no capacity, retry in a few seconds”, and matchmaking holds the match with players told it is finding a server. It does not spill into another region by default, because a 150 ms match is worse than a 10 s wait; a neighbouring region within everyone’s latency limit is a fallback after a longer wait. Meanwhile the fleet starts processes on spare hosts (~30 s) and the cluster adds hosts (~3 min).
3. How do you ship a hotfix when matches last 25 minutes?
   Blue-green per Fleet: create the new build’s Fleet with a Ready buffer, switch the allocator’s selector to the new build, and scale the old Fleet’s Ready count to zero. Old Allocated servers finish naturally, and within ~25 minutes the old Fleet is empty. Clients must be on a compatible build, so the client patch and the selector switch go together, with matchmaking requiring the right build on each ticket. Ending old matches early is possible for a severe bug, as a product decision with compensation.
4. How do you forecast capacity for a season start?
   Take the same weekday’s concurrency curve per region, scale it by the ratio seen at the last few launches, adjusted for pre-registrations, and add a margin. Raise fleet minimums an hour ahead, pre-pull the new image to every host, and book burst capacity from the cloud provider days ahead, since a region can run out of one instance type when several big launches land on the same morning. Reactive buffers then absorb the forecast’s error.
5. What do you measure to know the fleet is healthy?
   Allocation latency p99 and failures per region; the Ready buffer expressed as minutes of demand (Ready count ÷ start rate); host utilisation and the share of hosts with no Allocated servers, which should be draining away; tick time p99 per host, where a noisy neighbour shows up first; and crashes per 1 000 matches per build.

Ask: Why not save progress to the database as the match goes, and why not replicate the match so a crash doesn’t lose it?

Good answers name: Count in memory, one result per player through a queue, compensate a crashed match, Write progress to the profile database during the match, Hot standby replica of every match, Periodic checkpoints of world state, restored on a new server.

Our pick: The game server counts everything in memory and, when a player’s match ends (eliminated, quit or won), publishes one signed result to Kafka keyed by account id with acks from all replicas. The post-match pipeline checks it against the roster from match.started and the playlist’s caps, then applies it through the profile service in one transaction on the player’s shard: insert (match_id, account_id) into applied_results, which is unique, then update XP, level, battle pass, quests and stats and insert item grants; a duplicate inserts nothing and stops. It commits its Kafka offset only after the transaction, so a crash causes a duplicate, never a loss. A crashed server is not failed over: Agones marks it Unhealthy within ~6 s, the allocator emits match.aborted for the players still alive, and the pipeline grants XP for minutes played, without stats or placement, through the same unique row. Crash rate per build is a release gate, which is what keeps losses under 1 in 1 000.

1. The profile store is down for 20 minutes at peak. What happens?
   Matches are unaffected: servers hand results to Kafka and carry on. About 2 k results a second accumulate, 2.4 M in 20 minutes, a few GB. Players see “rewards pending”. When the store recovers, the pipeline drains the backlog at a rate capped so it does not starve logins, which takes minutes; nothing is lost and nothing applies twice.
2. Why key the queue by account id rather than match id?
   Because the write lands on the account’s shard, and ordering matters per account, not per match: two results for the same player after a quick re-queue should apply in order so a level-up reward is granted once, at the right tier. Keying by match would spread one partition’s writes across every shard and let one player’s results race each other.
3. A whole host dies. What is different?
   Twenty matches abort at once instead of one: ~2 000 players ask the gateway about their match within seconds, and the pipeline gets ~1 000 aborted records. Both are small loads. The thing to watch is correlated failure: if hosts die together (a zone, a bad kernel update), that region’s Ready buffer drops too, so clusters span zones and host maintenance is staggered.
4. A bug in the new build crashes 5 % of matches at minute 20. What happens?
   The crash-rate-per-build alert fires within minutes, while the rollout is still a small share of the fleet. The rollout halts and the allocator’s selector goes back to the old build for new matches; servers on the bad build keep running until they finish or crash. Compensation runs for every aborted match automatically. This only works if clients can play on either server build, which is why client and server compatibility is versioned.
5. Could players farm the compensation?
   Only by crashing servers on purpose, which would be an exploit to fix urgently. Compensation is minutes played times the playlist’s average XP rate, so it is never better than playing, and it is capped per account per day. Accounts that turn up in crashed matches far more often than chance are flagged for review.

Ask: A new season starts at 9 a.m. and 3 million players open the game within ten minutes. What keeps login, the lobby and the store up?

Good answers name: Waiting room with an admission rate driven by backend health, Autoscale the backend to meet the spike, Return 503 and rely on client back-off, Stagger the release by region or account cohort.

Our pick: A waiting room in front of login, per region. Joining is an INCR on a tail counter in Redis, which yields a ticket signed with its number; an admission controller advances an admitted watermark by R every second, and a poll whose number is below the watermark gets a short-lived signed admission token that the gateway checks without calling anyone. R is a control loop on backend health: it grows ~5 % a minute while the profile store’s p99 and error rate are healthy and halves when they are not, starting low (~2 k/s) while caches warm. Position is ticket minus watermark and the estimate is position ÷ R; polls back off with position and are answered from a watermark cached on each queue node, so 3 M waiting players cost almost nothing. Ahead of the start, fleets and stateless tiers are pre-scaled from the forecast, and the lobby sheds optional parts (presence, news) before essential ones. The store is never queued or eventually consistent: a purchase is one transaction on the player’s shard keyed by a client purchase id, debiting the wallet ledger only if the balance covers it and granting the item in the same commit.

1. A queue node crashes. Does everyone lose their place?
   No. The ticket is signed and carries its number, so any node can answer a poll; the only shared state is two counters per region. If Redis itself is lost, the controller restores the watermark from its last logged value and sets the tail above the highest ticket number seen. A few players may be admitted slightly out of order, which is acceptable for a waiting room.
2. A player double-taps buy on a slow connection. What happens?
   The client generates one purchase id when the player confirms and reuses it on every retry. The second request hits the unique purchase id, finds the first transaction committed, and returns its result: one debit, one item. If the first never reached the database, the second performs the purchase. A new id is minted only when the player starts a new purchase, the same contract as a payment idempotency key.
3. A platform refund arrives for currency the player has already spent.
   Post a reversing ledger entry. The balance may go negative, and a negative balance blocks further purchases until topped up. Whether to revoke items already bought with that currency is a policy decision within each platform’s rules; repeat abusers are banned. The ledger keeps the full history auditable, the same reasoning as a payment ledger.
4. How do you choose the starting admission rate?
   From load tests and the last launch: the rate at which the profile store’s p99 stayed under target with cold caches. Start below it and let the control loop climb. The most valuable rehearsal before launch day is a game day that replays a recorded login surge against a production-sized staging environment to find the real limit.
5. The daily item shop changes at midnight UTC and everyone opens it at once.
   The catalogue is the same for everyone, so render it once and serve it from a CDN keyed by the rotation time, with clients fetching it with jitter over a minute. Only purchases reach the store service, and those are spread out by human decision time. Personalised offers are precomputed before midnight so the per-player part is off the critical minute.