Design Zoom

Real-time audio, video and screen share for 2 to 1,000 people per meeting, 300 M participants a day, under 150 ms mouth to ear.

Last updated 2026-10-26. Difficulty: hard. Patterns: webrtc, sfu, real-time-media, congestion-control, multi-region. Reported at Zoom, Google, Microsoft, Meta, Amazon, Discord.

Functional requirements

• Scheduled and ad-hoc meetings. A meeting id and link exist ahead of time (one-off or recurring), or are created on the spot. The live session exists only while someone is in it; most meetings are 2 to 5 people.
• Join from a browser or an app. Browsers use standard WebRTC with nothing to install; native apps speak the same protocols with hardware codecs and their own tuning. Join by link, with an optional passcode or waiting room.
• Audio and video for 2 to 1,000 participants. Anyone can unmute. Speaker view (one large tile) and gallery view (up to 25 tiles). A receiver on a weak link gets lower quality without affecting anyone else.
• Screen share. Text must stay sharp: high resolution at a low frame rate (5 to 15 fps), and it outranks camera video when bandwidth is short.
• In-meeting chat. To everyone or to one person, in one order for all participants, with recent history shown to late joiners and saved with the meeting.
• Host controls. Mute, remove, lock, admit from the waiting room, spotlight a speaker. Several people change this shared state at once, so it needs a single order.
• Cloud recording. Speaker view with the screen share, plus audio-only, ready within minutes of the meeting ending. Everyone in the meeting can see that it is being recorded.
• Out of scope. Phone dial-in and SIP room systems (an audio-mixing gateway that joins as a participant), webinars beyond 1,000 attendees (switch to live streaming over HLS), end-to-end encryption, live captions and AI summaries, billing.

Non-functional requirements

• Mouth-to-ear latency (< 150 ms in region). ITU-T G.114 puts comfortable conversation under about 150 ms one way. Capture, encode, jitter buffer and decode use ~80 ms, leaving ~70 ms for the network and the server. This is the requirement that drives the hardest trade-off: no buffering and no CDN, UDP instead of TCP, losses concealed rather than awaited, and a media server within ~20 ms of most users. Across continents physics sets the floor, ~235 ms New York to Singapore.
• Scale (300 M participants/day · 20 M concurrent). About 4 M live meetings at peak and ~40 k joins a second at the top of the hour. Most meetings are tiny; a few have 1,000 people.
• Quality adapts per receiver (audio before video). One person on a weak link must never degrade anyone else, and when bandwidth runs out, video degrades first. Audio should stay intelligible at around 10 % random packet loss.
• Join time (p90 < 2 s to first audio). REST join, WebSocket, SDP exchange, ICE and DTLS each cost a round trip or two, so servers must be close to users and the join path must survive the top-of-hour spike.
• Connectivity (works with only TCP 443 open). Some corporate, hotel and school networks block UDP or force an HTTPS proxy. Joins must still succeed there, at somewhat worse quality.
• Resilience (media back < 5 s after an SFU fails). A media server crash costs seconds of frozen media, never the meeting: roster, chat and host state survive, and clients reconnect on their own.
• Security and privacy (encrypted in transit · admitted users only). DTLS-SRTP on every media hop and TLS for signaling; only participants the meeting service admitted can connect to its SFU. Recordings are encrypted at rest, visible only to the account, and deleted on its retention schedule.

Back-of-envelope estimates

• Concurrent participants at peak: ~10 M avg · ~20 M peak. 300 M participant sessions a day × 48 minutes ÷ 1,440 minutes ≈ 10 M in meetings at any moment. Business hours in the Americas and Europe overlap, so the peak is about 2×: 20 M.
• Concurrent meetings at peak: ~4 M. 20 M participants ÷ ~5 per meeting = 4 M live meetings. The meeting, not the participant, is the unit the allocator places, so this is the number it juggles.
• Join rate at the top of the hour: ~3.5 k/s avg · ~40 k/s spike. 300 M ÷ 86,400 ≈ 3.5 k/s on average. But meetings start on the hour: if the peak hour has 25 M joins and 30 % land in the 3 minutes around :00, that is 7.5 M ÷ 180 s ≈ 42 k/s. The join API, the allocator and the SDP exchange are sized for the spike, not the average.
• One sender’s uplink: ~2.2 Mbps · ~275 packets/s. Simulcast 180p 150 kbps + 360p 500 kbps + 720p 1.5 Mbps + Opus audio 32 kbps ≈ 2.2 Mbps. Video in ~1,200-byte packets is ~225 packets/s, plus 50 audio packets/s (20 ms frames). That is ~45 % more than 720p alone, which is why the SFU pauses layers nobody is watching.
• One receiver’s downlink: ~2.8 Mbps (vs ~150 Mbps unbounded). One large tile at 720p 1.5 Mbps + 8 thumbnails at 180p 8 × 150 kbps = 1.2 Mbps + top-3 audio ~0.1 Mbps ≈ 2.8 Mbps, whatever the meeting size. Forwarding all 999 others as thumbnails in a 1,000-person meeting would be ~150 Mbps per receiver and ~150 Gbps for the meeting. Selection is the design, not an optimisation.
• Peak SFU egress: ~40 Tbps. Many participants are audio-only, camera-off or in two-person meetings, so the average downlink is ~2 Mbps: 20 M × 2 Mbps = 40 Tbps. Spread over ~20 regions, that is ~2 Tbps per region.
• SFU servers: ~4,000 at full load · ~6,000 with headroom. A forwarding server does ~10 Gbps of egress, about 1 M packets/s at 1,200 bytes, each packet encrypted for its own receiver. 40 Tbps ÷ 10 Gbps = 4,000 servers; run at 70 % to absorb failures, ~6,000. That is ~3,500 participants on a typical server, and a crash moves all of them at once.
• Recording storage: ~500 TB/day. If 1 M meeting-hours a day are recorded and the composite runs at ~1.1 Mbps (≈ 0.5 GB per hour), that is 500 TB/day, ~180 PB a year before retention. Raw per-track chunks are 2 to 3× larger but are deleted a week after the composite succeeds.

Components

• Meeting client (browser (WebRTC) · desktop · mobile): Captures and encodes one audio stream and three simulcast video layers, and decodes the handful of streams the SFU forwards to it. Keeps a signaling WebSocket and a separate media connection, so either can fail without the other. Browsers use the standard WebRTC stack; native apps can use hardware encoders and tune congestion control and reconnects themselves.
• Edge gateway (REST + WebSocket · TLS 443): Terminates TLS for the REST API (schedule, join, recordings) and holds each participant’s signaling WebSocket. Routes in-meeting messages to the instance that owns the meeting and fans events from it back out to the participants it holds. Never carries media, so a gateway restart costs a reconnect, not a dropped call.
• Meeting service (one owner per live meeting): The control plane of a live meeting: admission and waiting room, roster, roles, mutes and host actions, chat order, recording state, and which SFUs carry the meeting. Each live meeting is owned by one instance through a lease with an epoch, so every change is serialised and a stale owner is fenced. It tells SFUs who may connect and what they publish; it never touches a media packet.
• Meetings DB (Postgres · sharded by meeting id): Durable meeting records: schedule and recurrence, settings (passcode, waiting room, allowed regions), host and account, plus chat logs and recording metadata. Read at join and cached per meeting, written rarely. The live session’s fast-changing state deliberately does not live here.
• Live meeting state (Redis · replicated): The hot state of every live meeting: owner lease and epoch, roster, current speaker, the last 100 chat messages, and the meeting-to-SFU map per region. Written on every change so a new owner can take over in seconds. A few kilobytes per meeting across ~4 M meetings at peak, so tens of gigabytes in total.
• SFU allocator (load- and RTT-aware): Knows every SFU’s health and load from one-second heartbeats and picks servers for meetings: the participant’s lowest-RTT region, then the server with the most headroom, never past ~70 % so a neighbour’s failure can be absorbed. Spreads large meetings across several servers and finds replacement capacity when a server dies. Its view is soft state, rebuilt from heartbeats within seconds of a restart.
• Event bus (Kafka · key = meeting id): Meeting lifecycle events (started, ended, recording stopped, media recovered) and quality telemetry, keyed by meeting id so one meeting’s events stay in order. Feeds the composite pipeline, usage reporting and quality dashboards. Nothing on the live media path ever waits on it.
• STUN / TURN relay (UDP · TURN over TLS 443): Fallback connectivity for networks that block UDP or only allow HTTPS through a proxy: relays the participant’s media to the SFU over TLS on port 443. Colocated with SFUs, so the extra hop is about a millisecond. Credentials are short-lived HMACs issued at join and relaying is restricted to SFU addresses, so it is not an open proxy; STUN on the same servers serves peer-to-peer calls.
• Edge SFU (selective forwarding · UDP): Receives each participant’s media once and forwards packets to subscribers without decoding, choosing per receiver which streams (last-N by voice activity) and which simulcast layer fit its bandwidth estimate. Terminates DTLS-SRTP per participant, answers NACKs from a one-second packet cache and rate-limits keyframe requests. About 10 Gbps and 1 M packets a second per server, run to ~70 %.
• Recording bot (receive-only participant): A headless client launched from a warm regional pool when a host starts recording. Joins the SFU like any participant, subscribes to every audio track, the active speaker at 720p and the screen share, and writes the frames to object storage in 10-second chunks without decoding. It appears in the roster, so everyone can see the meeting is being recorded.
• Recording storage (S3 / GCS · encrypted): Raw per-track chunks with a manifest, and the finished outputs (MP4, audio-only, HLS) per account. Raw tracks are deleted a week after a successful composite; finished recordings follow the account’s retention policy. About 500 TB of new finished video a day at 1 M recorded hours.
• Remote-region SFU (same meeting, other region): An SFU in another region serving that region’s participants of the same meeting. Exchanges streams with the meeting’s other SFUs over the private backbone, subscribing once per stream on behalf of all its local receivers, and shares audio levels so every region agrees on who is speaking.
• Composite + transcode (batch · parallel by segment): Consumes recording.stopped, decodes and aligns the raw tracks by wall-clock time, renders speaker view with the share, and encodes MP4, audio-only and HLS renditions. Splits long recordings into 5-minute pieces encoded in parallel, so an hour is ready in minutes. Idempotent on recording id, so a redelivered event never produces a second file.
• Playback CDN (recordings only): Serves recorded meetings as HLS with signed, expiring URLs, pulling from recording storage on a miss. Live meetings never touch it: real-time media has no time for a cache. Deleting a recording purges it here too.

User flows

1. Join a meeting and start talking. Signaling and media take different paths: REST and a WebSocket set the meeting up, then audio and video go over UDP straight to an SFU in the participant’s nearest region. A strong answer knows what each round trip on the way to first audio is for.
   1. The client asks to join with the meeting id and passcode, and gets a short-lived join token. The meeting service reads the meeting’s settings (passcode, waiting room, allowed regions, host) from the Meetings DB, cached per meeting. The token is a signed JWT valid for 60 seconds and bound to (meeting, user), so it cannot be replayed into another meeting. The response also carries TURN credentials, so the fallback costs no extra round trip if UDP turns out to be blocked.
   2. The client opens a WebSocket and sends join; the instance that owns this meeting admits it. Each live meeting has exactly one owning meeting-service instance, recorded as a lease in the live state store, and the gateway routes every message for that meeting there. One owner serialises joins, mutes and host actions, so two co-hosts clicking "mute all" and "unmute" never apply in different orders on different clients. With a waiting room, the participant waits here until a host admits them.
   3. The meeting service finds the meeting’s SFU in the participant’s region, asking the allocator to place one if there is none. The region is the lowest-RTT one from the client’s probes, not GeoIP, which is often wrong for VPN, corporate and mobile users. The allocator picks the server with the most headroom from the heartbeats it receives every second and stops placing on a server at ~70 % of its egress or packet budget. The assignment is written to the live state store under the meeting’s epoch.
   4. The client sends an SDP offer with its audio and three video layers; the SFU answers through the meeting service. The offer lists Opus for audio and one video track with three simulcast encodings (rids q, h, f), plus the header extensions the SFU routes on: audio level, transport-wide sequence numbers and layer ids. The SFU is ICE-lite: its answer carries only its own public UDP and TCP 443 candidates and its DTLS certificate fingerprint. All tracks share one transport (BUNDLE) and RTCP rides on the same port.
   5. ICE connectivity checks reach the SFU over UDP, and a DTLS handshake on the same port derives the SRTP keys. A STUN binding request carrying the ICE credentials proves the path works and that this is the client the SFU was told to expect. DTLS then takes one or two round trips, so on a 20 ms path the whole exchange is well under 200 ms. Consent checks repeat every few seconds and keep the NAT binding open.
   6. If UDP is blocked, ICE falls back to TCP 443 at the SFU or to a TURN relay over TLS. The client tries UDP first and starts the TCP and TURN candidates in parallel after a short delay, so a blocked network costs about a second rather than a timeout. TURN over TLS on 443 looks like HTTPS to a corporate proxy. The relays sit in the same sites as the SFUs, so they add a millisecond; the real cost is that this participant gets TCP’s behaviour on loss.
   7. The client starts sending; the meeting announces the new participant and others start receiving it. Audio is Opus at ~32 kbps in 20 ms packets; video is 180p at ~150 kbps, 360p at ~500 kbps and 720p at ~1.5 Mbps, about 275 packets a second in all. Other clients update their layout from the roster event and the SFU starts forwarding each of them the layer they can use. Typical join to first audio is about a second.
2. A receiver’s bandwidth drops mid-meeting. The core read path: the SFU, not the sender, decides what each receiver gets. When one participant’s Wi-Fi degrades, only that participant’s streams get smaller; nobody else in the meeting notices.
   1. The client tells the SFU what it is showing: one large tile and eight thumbnails. The layout sets the ceiling: a 180-pixel-high thumbnail never needs the 720p layer, whatever the bandwidth. Constraints go on a data channel over the same DTLS connection, so they reach the SFU in one hop instead of through signaling.
   2. The receiver sends congestion feedback, and the SFU estimates its downlink. Every packet the SFU sends this receiver carries a transport-wide sequence number, and the receiver reports when each arrived every 50 to 100 ms. The SFU runs a delay-based estimator: when packets arrive further apart than they were sent, a queue is building on the path, so it backs off before loss starts. Loss is the backstop signal, not the main one.
   3. The estimate falls from 3 Mbps to 1.2 Mbps; the SFU re-plans this receiver’s streams in priority order. Audio for the top three speakers first (~100 kbps), then the large tile, then thumbnails. 720p plus eight 180p thumbnails is ~2.8 Mbps, so the large tile drops to 360p (~500 kbps) and the thumbnails drop to 7.5 fps by forwarding only their base temporal layer (~60 kbps each): about 1.1 Mbps in all. Nothing changes for any other receiver.
   4. To switch the large tile to 360p, the SFU waits for a keyframe on that layer, asking the sender for one if needed. Simulcast layers are independent streams, so a receiver can only start decoding a new layer at a keyframe. The SFU sends a picture loss indication for that layer at most once every ~500 ms, so fifty receivers switching at the same moment trigger one keyframe, not fifty. Until it arrives, the receiver keeps getting the old layer, so nothing freezes.
   5. Video packets lost on the last hop are re-sent from the SFU’s own cache. The receiver NACKs each missing video packet; the SFU keeps the last second of packets per stream and retransmits from there, so the repair crosses only the last leg instead of going back to the sender. That only helps when the round trip is under ~100 ms; beyond that the receiver asks for a keyframe instead. Audio is not retransmitted: Opus FEC and loss concealment cover it.
   6. When the link recovers, the estimate climbs slowly and layers step back up one at a time. Going up is deliberately slower than going down: the SFU probes for spare capacity with padding before committing to a bigger layer, and waits a few seconds between steps. Flapping between 360p and 720p every second looks worse than staying at 360p.
3. A 1,000-person all-hands on three continents. The scale-breaking case. Forwarding everything to everyone would be 150 Mbps per receiver; last-N selection, cascaded regional SFUs and one owner for meeting state keep each receiver near 3 Mbps and each stream crossing an ocean once.
   1. The meeting gets SFUs in each region its participants are joining from. The invite list says ~1,000, so the allocator spreads the meeting across several SFUs per region instead of packing it onto one: a single server could carry the ~2.8 Gbps, but losing it would drop everyone at once. Each SFU is told which other SFUs belong to the meeting.
   2. Regional SFUs link over the backbone and subscribe on behalf of their local participants. The London SFU requests each remote stream once, at the highest layer any of its local receivers needs, and fans it out locally. Nine video senders at up to three layers is about 20 Mbps per link per direction, small next to the ~2.8 Gbps delivered to receivers inside the regions. The backbone has far less loss and jitter than the public internet, so the extra hop costs a few milliseconds and buys stability.
   3. Audio levels from every SFU produce one meeting-wide speaker ranking; each receiver gets the top three voices and the last nine speakers’ video. Clients tag every audio packet with its level, so ranking needs no decoding. SFUs share their local senders’ levels a few times a second and all compute the same order, with about a second and a half of hysteresis so a cough does not steal the large tile. The other 990 participants cost almost nothing: muted microphones with discontinuous transmission send a few packets a second, and nobody subscribes to their video, so their layers are paused.
   4. The CEO shares her screen, and it becomes the highest-priority video for every receiver. Screen content is encoded for sharp text at a low frame rate, typically 1080p or more at 5 to 15 fps and 1 to 2 Mbps, with the encoder tuned for detail rather than motion. The SFU allocates it right after audio, so a receiver on a weak link sees readable slides and a smaller speaker rather than the reverse.
   5. Someone asks a question in chat; the meeting’s owner sequences it and fans it out to 1,000 people. One owner per meeting gives chat a single order: it assigns the next sequence number, appends the message to the meeting’s chat log, and publishes it to the gateways holding this meeting’s participants, which write it to each WebSocket. One message becomes 1,000 writes spread over the gateways, so chat is rate-limited per participant and hosts can restrict who may message everyone.
   6. A late joiner gets the current state in one message, then receives only the active streams. The join response carries the roster, who is sharing, the current speaker and the last 100 chat messages from the live state store, so the newcomer never replays the meeting’s history. Roster updates to everyone else are batched every few hundred ms during join spikes: 1,000 people joining at 10:00, each announced to everyone, would otherwise be a million individual messages.
4. An SFU dies mid-meeting. The failure path. Media state on the SFU is lost, but meeting state never lived there, so recovery is a media reconnect over the still-open signaling connection, not a rejoin. Target: voices back in about three seconds.
   1. The SFU stops heartbeating, and the allocator declares it dead when its lease expires. SFUs report health and load every second with a 3-second lease. Clients notice independently: no RTP or RTCP for ~2 seconds while their WebSocket is still up gets reported to the meeting service, which also catches a hung process whose heartbeat thread still runs.
   2. For each affected meeting, the owner gets replacement capacity spread across the region and records the new assignment. The dead server carried ~680 meetings and ~3,400 participants. The allocator spreads them across many servers so no single neighbour takes the whole load, which is what the 30 % headroom is for. The new mapping is written with a bumped epoch, so any late message based on the old assignment is ignored.
   3. The meeting service tells each participant over signaling to reconnect its media. The WebSocket goes to a gateway, not to the SFU, so it survived the crash. That is the whole point of keeping signaling and media on separate paths.
   4. Each client performs an ICE restart against the new SFU, publishing audio before video. New ICE credentials and a new DTLS handshake, then the client republishes its tracks and re-subscribes from the roster it already holds. Clients add up to a second of random jitter, so 3,400 reconnects arrive spread out instead of in one burst. Audio first means voices return before video does.
   5. Receivers need keyframes for every video stream; the new SFU deduplicates the requests. Every receiver’s decoder lost its reference frames, so each needs a keyframe from every sender it watches. The SFU sends one keyframe request per sender layer and holds the rest; otherwise a 200-person meeting would ask each speaker for 200 keyframes in the same second.
   6. The recovery is recorded per participant, as time from failure to first audio. The target is voices back in ~3 seconds at p50 and under 5 at p99: about 2 seconds to detect and 1 to reconnect. The events feed dashboards and the postmortem; a slice of participants that took 20 seconds usually points at one client version with bad reconnect logic.
5. Cloud recording, from bot to playback. The asynchronous path. A bot joins like any participant and writes raw tracks to object storage; the expensive decoding and encoding happen after the meeting on batch capacity, so recording never slows the call down.
   1. The host starts recording; the meeting service launches a recording bot near the meeting’s SFU. Recording is a meeting state change, applied by the owner and broadcast so every client shows the recording banner. The bot comes from a warm pool in the same region, so it is in the meeting within a couple of seconds.
   2. The bot joins the SFU as a receive-only participant and subscribes to every audio track, the speaker’s video and the screen share. To the SFU the bot is one more subscriber, so there is no new code in the forwarding path. It asks for the 720p layer of whoever is on stage and the full-resolution share, whatever other receivers get. Separate audio tracks let the composite balance levels and let a transcript say who said what.
   3. The bot writes each track to object storage in 10-second chunks, without decoding. Frames are depacketised into a container, with RTP timestamps mapped to wall-clock time from RTCP sender reports so tracks can be aligned later, and a manifest lists chunks and gaps. A crashed bot loses at most the chunk in progress, and its replacement continues with a new chunk sequence.
   4. When recording stops, the meeting service publishes an event and a composite job picks it up. Compositing is batch work: it can wait seconds for capacity and run on cheaper or spot machines. The event carries the manifest location, and the job is idempotent on recording id, so a redelivered event never produces two files.
   5. The job aligns and decodes the tracks, renders speaker view with the share, and encodes MP4, audio-only and HLS. An hour-long recording is split into 5-minute pieces encoded in parallel, so it is ready in minutes rather than in an hour. Output runs at ~1.1 Mbps, about 0.5 GB per hour; at 1 M recorded hours a day that is ~500 TB of new storage daily, which is why retention defaults matter.
   6. The host opens the recording and the player streams it from the CDN with a signed URL. Recorded playback is ordinary video on demand: HLS segments cached at the edge and adaptive bitrate in the player. The URL is signed, expires within hours and is issued only after checking the account’s sharing settings, so a leaked link stops working. Deleting a recording removes the objects and purges the CDN.

Deep dives

The theory behind it

• Video calls and WebRTC: How Zoom-style video calling works: WebRTC media and signaling, NAT traversal with STUN and TURN, peer-to-peer versus SFU versus MCU, simulcast and adaptive quality, scaling large meetings with cascaded SFUs, recording, latency budgets and reliability.
• Networking for system design: DNS, TCP and UDP, HTTP/1.1 versus HTTP/2 and HTTP/3, TLS, proxies and reverse proxies, and the latency numbers behind them, at the depth a system design interview expects.
• Global traffic management, DNS and anycast: How users reach the nearest healthy region: DNS resolution and TTLs, GeoDNS and latency-based routing, anycast, global load balancers, health checks and failover, the limits of DNS-based failover, and steering traffic during incidents and migrations.

Related

• All 83 system design interview questions
• Run this question for a friend: the interviewer kit