Design Discord

• Servers (guilds) with text channels: A guild has channels grouped in categories and can have 1 M+ members. DMs and group DMs are private channels outside any guild and work as in Slack.
• Send and receive messages in real time: Text, mentions, replies, edits and deletes. Sent over REST, received over one gateway WebSocket per client.
• History that scrolls back years: Newest page on open, then pages before a message id; jump to a message link.
• Roles and permissions: Roles carry permission bitfields; channels override them per role or per member. Decides who sees a channel, who can send, who can ping @everyone. Slowmode per channel.
• Presence and typing: Online, idle, do not disturb, offline, plus activities such as "Playing X". Typing shows in the channel you have open.
• Member list: The sidebar of who is online, grouped by role. Must work in a guild with 100 k people online.
• Unread indicators and mention counts: Per user per channel, the same on every device. @everyone and role pings count too.
• Voice channels: Join a channel and talk with everyone in it. Video and screen share ride the same servers and are not designed separately here.
• Bots: Bots connect to the gateway like users. A bot in a million guilds must split its connection into shards.
• Out of scope: Search, push notifications, the file upload pipeline, AutoMod and moderation (see Content Moderation), Nitro and payments, the SFU internals (see Zoom).

• Scale (150 M MAU · 20 M concurrent): Concurrency sizes the gateway fleet; 4 B messages a day size storage. Neither is the hardest number in the design.
• Largest guild (1 M members · 100 k online): The requirement that drives the hardest tradeoff. One process orders a guild’s events, so everything that scales with online sessions has to be moved out of it or cut down to what each client displays.
• Delivery latency (p99 < 500 ms): From send to every session viewing the channel in the region, in a 100 k-online guild too. Sessions not viewing it get badges within seconds rather than every message.
• Ordering (total order per channel): Snowflake ids sort by time; each guild applies its events in one order, so a role change and a message never race.
• Durability (no loss after 200 OK): A message is written at quorum before it is acknowledged or delivered. The live stream may drop an event, because history is the source of truth.
• Availability (99.99 % messaging): Presence, typing and member lists degrade first, on purpose. Voice runs on its own fleet, so a chat incident does not cut calls.
• Voice latency (< 200 ms mouth to ear): Within a region. Drives regional voice servers and media over UDP.
• Permissions (checked at send, read and fan-out): A member who loses a role stops receiving that channel from the next event on. A hidden channel never leaks through history, mentions or the member list.

• Messages per second: ~46 k avg · ~115 k peak. 4 B messages/day ÷ 86 400 s ≈ 46 k/s. Evenings run about 2.5× the average, so ~115 k/s at peak. One partition takes thousands of writes a second, so writes are not the hard part.
• Message deliveries per second: ~4.6 M avg · ~12 M peak. DMs reach one or two sessions and big guilds thousands, but delivering only to sessions that have the guild or the channel open keeps the average near 100 sessions per message: 46 k × 100 = 4.6 M deliveries/s, ~12 M at peak. Each one is filtered by permission first.
• Gateway nodes: ~400. 20 M concurrent sessions at peak. A session holds its guild subscriptions and a replay buffer budgeted at ~100 KB, so 50 k sessions is ~5 GB per node with CPU left for compression and delivery: 20 M ÷ 50 k = 400 nodes.
• Presence deliveries, naive: ~170 M/s. 20 M users each changing status or activity about every 10 minutes is 20 M ÷ 600 = 33 k changes/s. Sending each change to the ~5,000 online members who share a guild with that user: 33 k × 5,000 ≈ 170 M/s, about 35× the message fan-out. Sending only to the ~50 sessions displaying the member brings it to ~1.7 M/s. Presence, not chat, is the biggest stream.
• One event to the biggest guild: ~200 ms of one core. 100 k online sessions × ~2 µs per send = 200 ms for a single event from one process. A big guild produces dozens of events a second (messages, joins, role and voice changes), so one process cannot fan out alone. Relays of up to 15 k sessions turn it into ~7 sends from the guild.
• Message storage per year: ~440 TB. ~300 B per message with ids, author, flags and attachment references (files live in object storage). 4 B × 300 B = 1.2 TB/day ≈ 440 TB/yr before replication, ~1.3 PB with three replicas, about 1.5 trillion messages a year.
• Busiest channel per day: ~520 MB. A channel sustaining 20 messages/s stores 20 × 86 400 × 300 B ≈ 520 MB/day, so a 10-day bucket would be a 5 GB partition. Keeping partitions under ~100 MB means a 1-hour bucket for that channel (~22 MB), while a typical channel fills a 10-day bucket with a few megabytes.
• Voice egress: ~360 Gbps. Assume 3 M people in voice. Opus at ~64 kbps plus packet overhead is ~80 kbps per stream, and a listener receives about 1.5 streams because clients send only while speaking: ~120 kbps and ~75 packets a second each. 3 M × 120 kbps = 360 Gbps, more bandwidth than all of chat.

• Discord client (desktop · web · mobile · bots): Holds one gateway WebSocket for everything pushed to it (messages, presence, member-list ranges, voice state) and uses REST for actions such as send, edit and history. Remembers the last sequence number it saw, so a dropped socket resumes instead of reloading. Sends a nonce with each message so a retried POST cannot post twice. A bot is the same client without a UI, and a big bot opens one connection per shard.
• API edge (REST · auth · rate limits): Terminates TLS, validates the token and applies rate limits before anything else runs: per-route buckets keyed by the route and its major id (channel, guild, webhook), a global limit per user or bot (Discord documents 50 requests a second for bots), and a ban for clients that keep sending invalid requests. Returns 429 with retry_after and bucket headers so good clients pace themselves. Stateless; counters live in a sharded in-memory store.
• API service (send · history · Snowflake ids): The write path for messages and the front door for history. On a send it asks the guild for the member’s permissions and slowmode, assigns a Snowflake id (42 bits of milliseconds, then worker, process and sequence), writes through the data service, returns 200, then hands the event to the guild for live fan-out and to the event log. It never fans out itself, so a message costs it the same in a 10-member guild and a 1 M-member one.
• Data services (Rust · routed by channel): A thin layer between the API and the messages store, one endpoint per query. Requests are routed by consistent hashing on channel_id, so every read of a channel lands on the same instance, and identical reads in flight at the same time are coalesced into one database query whose result goes to every waiter. Discord has written about building exactly this in Rust in front of ScyllaDB to tame hot partitions.
• Messages store (ScyllaDB · (channel, bucket)): Wide-column store holding every message: partition (channel_id, bucket), clustering message_id descending, with the bucket computed from the Snowflake’s timestamp. Buckets default to 10 days, the width Discord described in 2017, and the busiest channels get narrower ones so no partition passes ~100 MB. LOCAL_QUORUM writes; about 440 TB a year before replication.
• Sessions & presence (user → sessions · status): Knows every live session of every user and folds them into one status (online beats idle) plus activities like "Playing X". Coalesces changes so a user broadcasts at most one update every few seconds, sends it to friends’ sessions, and hands it to the user’s guilds, which decide who actually sees it. Also the directory for events aimed at one user, such as a read ack to their other devices.
• Event log (Kafka · key = channel_id): Durable stream of message events for consumers that must not miss one: mention counting in read states, plus search, AutoMod and push, which are out of scope here. Keyed by channel_id so a channel’s events stay in order. Live delivery deliberately does not go through it: a consumer hop adds latency, and a missed live event is recovered from history anyway.
• Read states (user × channel · LRU): One record per user per channel: the last message id read, a mention count and a few flags. Held in memory in an LRU sharded by user_id, with each change committed to a wide-column table about 30 seconds later or on eviction, which is how Discord has described its own read states service (rewritten from Go to Rust to remove GC pauses). Kept apart from messages because it is per user and changes on every read, while a message is written once.
• Gateway (WebSocket · ~50 k sessions/node): Holds the persistent WebSockets. Each session has an id, a sequence number stamped on every event it is sent, and a bounded replay buffer, so a client that drops can RESUME with its last sequence number and get only what it missed. On identify it subscribes the session to each of its guilds using a local copy of the guild ring, tracks which guild and channel the client is looking at, and compresses the stream. Bots connect here too, one session per shard.
• Guild actors (one process per guild): Each guild is one in-memory process that owns its state (channels, roles, overwrites, members, voice states) and its connected sessions, so every event in the guild is applied in one order. It caches each role set’s computed permissions per channel, filters every event by VIEW_CHANNEL, and sends one batch per gateway node rather than one message per session. Placed on guild nodes by a hash ring on guild_id; a dead node’s guilds restart elsewhere from the guilds DB.
• Guilds DB (keyed by guild_id): Durable home of guild state: guilds, channels, roles with their permission bitfields, channel overwrites, members and their role ids. Guild actors load from it on start and write through on every change, so a restarted actor rebuilds exactly. A 1 M-member guild is about a million member rows of ~100 bytes, ~100 MB, which loads in seconds.
• Voice servers (regional SFUs · UDP): Selective forwarding units in many regions: each speaker sends Opus packets once and the server forwards them to everyone else in the voice channel without mixing. One server hosts a channel’s call, chosen by the guild from the least-loaded healthy servers in the region nearest the participants. Signalling is a WebSocket to the server, opened with a token the guild hands out over the gateway; media is UDP and never peer to peer, which keeps users’ IP addresses private.
• Fan-out relays (≤ 15 k sessions each): For guilds too big for one process to fan out, the guild hands its session list to relays, each owning a slice of up to about 15,000 sessions, the figure Discord has published. The guild sends an event once per relay; each relay filters its slice (who is viewing the channel, who may see it, who is passive) and batches by gateway node. Relays also take session joins and leaves, so the guild process stays free for state changes.
• Service discovery (etcd · guild ring · voice load): Small, strongly consistent registry of which nodes are alive. It holds the hash ring that places guild processes on guild nodes, which every gateway and API node keeps a local copy of and watches, and the health and load of every voice server, which the guild reads when it allocates a call. Read from local caches, so it is never on the per-message path.

1. Send a message in a channel: The core path. The message is durable before the sender sees it posted, and every session that should see it gets it within half a second, with permissions checked twice: once to send, once per recipient.
   1. Client POSTs the message with a nonce; the edge authenticates it and applies rate limits.
   2. API service asks the guild actor whether this member may send here and whether slowmode allows it.
   3. API assigns a Snowflake id and writes the message through the data service.
   4. API returns the stored message to the sender.
   5. API dispatches MESSAGE_CREATE to the guild actor, which picks the recipient sessions and sends one batch per gateway node.
   6. Each gateway node stamps the next sequence number for each session, buffers the event and writes it to the socket.
   7. API publishes the event to the log; the read-state service raises mention counts for the users it pings.
2. Open a busy channel and scroll back: The main read path. Opening a channel is a REST read of the newest page; the hard case is thousands of people opening the same channel within seconds, which would aim a storm of identical reads at one partition.
   1. Client tells its gateway session which guild and channel it is now looking at.
   2. Client asks for the newest 50 messages; the API checks VIEW_CHANNEL and READ_MESSAGE_HISTORY with the guild.
   3. API calls the data service instance that owns this channel on the hash ring.
   4. If an identical query is already in flight, the request joins it; otherwise the data service reads the newest bucket.
   5. Scrolling back asks for messages before the oldest id on screen; the service walks buckets backwards, skipping empty ones.
   6. Client acks the newest message it has read; the read-state service moves the marker and clears the mention count.
   7. The ack reaches the user’s other devices so their badges clear too.
3. An @everyone in a 1 M-member guild: The scale-breaking case: 100,000 members online, one message that pings them all, and 30,000 of them opening the channel within a minute. Every per-session cost is multiplied by 100,000, so the design decides who gets what.
   1. A moderator posts an announcement with @everyone; the API checks MENTION_EVERYONE and stores it as usual.
   2. The guild actor, which hands fan-out to relays above a size threshold, sends the event once to each of its seven relays.
   3. Each relay filters its slice: the full message for sessions viewing the channel, a mention badge for everyone else who can see it.
   4. Mention counts are not written per member: the read-state service records one @everyone mention for the channel.
   5. 30,000 members open the channel in the next minute; their history reads coalesce at the data service.
   6. Opening the guild, each client subscribes only to the member-list ranges it is showing.
   7. A member’s presence changes; the guild forwards it only to sessions whose subscribed ranges show that member.
4. A dropped socket, then a dead gateway node: Phones lose signal constantly, so reconnecting must be cheap: a resume replays only the missed events. When a whole gateway node dies its sessions are gone, and 50,000 clients must identify again without stampeding the guild tier.
   1. A phone switches networks and its socket closes; the gateway keeps the session and keeps buffering its events.
   2. The client reconnects to its resume URL and sends RESUME with its session id and last sequence number.
   3. If the gap is older than the buffer holds, the gateway refuses the resume and the client identifies from scratch.
   4. Later the gateway node itself crashes; guilds and relays see its sessions vanish and drop them.
   5. Clients’ resumes fail, so they back off with jitter and identify on other nodes through an admission queue.
   6. The new node subscribes the session to each guild via its cached guild ring; a guild with nobody online is loaded from the guilds DB first.
   7. The gateway loads the user’s read states and sends READY, then a GUILD_CREATE as each guild answers.
   8. Presence hears the user is online again but broadcasts nothing, because the status never changed.
5. Join a voice channel and talk: Voice runs on its own fleet: the gateway carries only signalling, and media goes over UDP to a voice server near the participants. Chat and voice fail independently.
   1. Client sends a voice state update over the gateway: join voice channel V.
   2. If V has no voice server yet, the guild picks the least-loaded healthy one in the nearest region.
   3. The guild records the voice state, tells the guild who joined, and sends this session the server’s address and a token.
   4. Client opens a signalling WebSocket to the voice server, then starts sending UDP media.
   5. The voice server forwards each speaker’s packets to everyone else in the channel without decoding them.
   6. The voice server dies; clients notice, the guild allocates a new one and the call carries on.

Ask: Why route every event in a guild through one process, and what happens when that guild has 100,000 people online?

Good answers name: Guild actor fans out directly, batched per gateway node, Actor owns state; relays own slices of sessions; active, passive and lazy delivery, Per-channel pub/sub topics that gateways subscribe to, Shard one guild across several actors by channel.

Our pick: One actor per guild, placed on guild nodes by a hash ring on guild_id, owns the state and decides the order of every event. Small guilds fan out directly, grouped by gateway node the way Discord’s Manifold library groups processes by node, so a send costs one message per node. Above about 15,000 online sessions the actor hands its session list to relays of up to 15,000 sessions each (Discord’s published figure) and sends each event once per relay, with state changes on the same ordered stream so a relay never filters a message with stale permissions. Delivery is tiered: sessions viewing a channel get its messages and typing; passive sessions get mention badges and a coalesced unread hint; the member list is sent only as the ranges each client shows. For a guild with 100,000 online that is about seven sends per event from the owner and roughly 10,000 full deliveries instead of 100,000. Discord has described this path in its write-up on Midjourney’s server: one Elixir process per guild, passive sessions, relays, and member data in ETS so other processes can read it.

1. The node running a big guild’s actor crashes. What happens?
   Sessions see the guild go unavailable, and the ring assigns it to the next node, which loads it from the guilds DB: a 1 M-member guild is about 100 MB of member rows and loads in seconds. Gateways resubscribe their sessions and clients get a fresh GUILD_CREATE. No message is lost, because every message was written to the store before dispatch; what is lost is a few seconds of live pushes, which clients refill by reading the newest page when the guild returns. Slowmode timers and typing are in memory and simply reset.
2. How do relays see guild state without asking the owner about every event?
   The owner sends state changes down the same ordered stream as events, so each relay keeps a replica that is always at least as new as the event it is filtering. On the owner’s own machine that replica can be a shared read-only table, which is what Discord does with member data in ETS; on other machines it is a copy fed by the stream. Every event carries the state version it was produced at, so a relay can assert it never filters with older permissions.
3. Why not make every session active and buy more hardware?
   Because the cost is multiplicative, not additive. A guild with 100,000 online producing 50 events a second is 5 M deliveries a second if everyone is active, the same as all message fan-out in the system on average. With 90 % passive and messages scoped to viewers of the channel, it is a few hundred thousand. It is also what users expect: a guild that is not on screen needs a badge, not every message.
4. Can two relays deliver the guild’s events in different orders?
   Each relay receives events from the owner in the owner’s order on one stream and delivers to its sessions in that order, so every session sees the guild’s events in the same order. Two sessions on different relays may see an event a few milliseconds apart, which nobody can observe. Within a channel clients also sort by message id.
5. How do you know a guild process is keeping up?
   Measure event age: the time from an event entering the owner’s queue to its dispatch, per guild, with an alert on the p99 of the largest guilds, plus queue length. When it grows, the owner sheds the least important work first: presence outside list ranges, then typing, then member-list updates, and messages last. The biggest guilds also get dedicated nodes, so a noisy neighbour cannot slow them.

Ask: A phone loses signal for 20 seconds. How does it catch up without reloading everything, and how does a bot that is in a million guilds connect at all?

Good answers name: Re-identify on every reconnect, Sessions with per-session sequence numbers, a bounded replay buffer and RESUME, A durable event log per user for unlimited resume, Per-channel cursors and REST sync, as in Slack.

Our pick: Every session has an id and a sequence number that the gateway stamps on each dispatch; the client sends its latest sequence in each heartbeat (about every 45 s, the interval Discord’s docs use as an example). The gateway keeps a per-session replay buffer bounded at about 2 minutes or 1 MB and keeps a disconnected session alive for that long. RESUME with (session_id, seq) replays the gap and ends with RESUMED; anything older gets INVALID_SESSION and a fresh identify, after which the client reloads the channels on screen from the store. Planned restarts drain a node by handing its sessions and buffers to another node and telling clients to reconnect there, so a deploy costs resumes, not identifies. Identifies are admitted at a fixed rate per node behind jittered client backoff, so a crashed node’s 50,000 sessions return over tens of seconds. Bots shard: a bot in 1 M guilds needs at least 400 connections at the documented limit of 2,500 guilds per shard, and each guild belongs to shard (guild_id >> 22) % num_shards, both as Discord documents them.

1. Why (guild_id >> 22) and not guild_id % n?
   The low 22 bits of a Snowflake are the worker id, process id and a per-process counter, which are not evenly spread: ids minted by one process in one millisecond differ only in the counter. Shifting them off leaves the creation time in milliseconds, which is effectively uniform across guilds, so the modulo spreads them evenly. It also means anyone can compute a guild’s shard from its id with no lookup.
2. A big bot restarts all 400 shards at once. What protects the guild tier?
   Identify limits. Discord documents 1,000 identifies a day by default and a max_concurrency of shards that may identify in each 5-second window, so a large bot’s restart takes minutes by design. Each shard that resumes instead costs nothing. Intents help too: a bot that has not asked for presences or member updates is never sent them, which is why those intents are privileged.
3. How much memory does the replay buffer cost?
   Most sessions get a few events a second, so a 20-second gap is a few kilobytes. Budget 100 KB per session for the bad case and 50,000 sessions is about 5 GB per node, which is fine. The cap is what matters: a session in a hyperactive guild that would pass 1 MB stops being resumable, which is the right trade against unbounded memory.
4. Sending is REST. Why not send messages over the socket?
   REST gives every action the same auth, rate limits, idempotency through the nonce and error codes, and lets the edge reject abuse before it reaches the gateway. The gateway then scales on connections and outbound bytes only. The socket carries the few client frames that are about the session itself and are cheap to lose: heartbeats, presence, focus, member-list ranges, read acks and voice state.
5. How do you deploy the gateway without logging everyone out?
   Drain a few percent of nodes at a time: stop placing new sessions on a node, move each session with its buffer to a successor, then send the client a reconnect so it resumes against the successor. The successor updates each guild with the session’s new node, which is far cheaper than a fresh subscribe. Only a crash forces identifies, and those are admitted at a controlled rate.

Ask: Where do trillions of messages live, and what happens when 30,000 people open the same channel in the same minute?

Good answers name: Wide-column (channel_id, bucket) with per-channel bucket width, behind data services that route by channel and coalesce, Partition by channel_id alone, Cache of the newest messages per channel in front of the store, Sharded relational database by channel_id.

Our pick: Messages live in a ScyllaDB-style wide-column store keyed ((channel_id, bucket), message_id DESC), where bucket is the Snowflake’s millisecond timestamp divided by the channel’s bucket width. The default width is 10 days; a channel whose rate would put more than ~100 MB in a bucket gets a narrower one (1 hour for the 20-a-second channels, about 22 MB), recorded in channel metadata with the message id it applies from, so old rows never move. Every read and write goes through data services routed by consistent hashing on channel_id, so one instance sees all of a channel’s traffic and merges identical in-flight queries: the first request runs it and the rest wait for its rows. A per-channel list of non-empty buckets lets scroll-back skip gaps. Read states and mention counts are separate, because they change per user and per read while a message is written once.

1. The data service instance that owns a hot channel dies mid-storm.
   The ring moves its channels to the next instance within a heartbeat. Waiters on in-flight queries get errors and retry, landing on the new owner, which starts coalescing them again. The store sees a short bump of uncoalesced reads, a few thousand, which three replicas absorb. Data services hold no durable state, so there is nothing to recover: they are a cache of work in flight.
2. How do edits and deletes work in an LSM store?
   An edit rewrites the row under the same key, a newer version that wins on read. A delete writes a tombstone, and a channel where millions of messages are deleted leaves millions of tombstones that every read must skip. Discord described exactly this in 2017: a channel with millions of deleted messages became slow to load, and they cut tombstone lifetime from 10 days to 2 and started tracking empty buckets. I would also let bulk deletes drop whole buckets where they can.
3. Why not compute unread counts from the messages table?
   It would be a count query per user per channel on every connect, across buckets, for 20 M concurrent users. Unread is a comparison: the channel’s last message id, which the guild keeps in memory, against the user’s last read id in read states. Mention counts are counters in read states fed from the event log, with @everyone as a short channel-level list. None of it touches the messages store.
4. How would you move trillions of messages to a new database without downtime?
   Put the data services in front first, so the API no longer knows which database it talks to. Then write new messages to both, backfill history with a bulk migrator reading token ranges in parallel, verify by comparing sampled reads from both, and switch reads over. Discord described doing this in 2022 with a Rust migrator that moved up to 3.2 M messages a second and finished the copy in nine days.
5. How do you know coalescing is working?
   Per data service instance, track requests in against queries out and the share of requests that joined an in-flight query, with the top channels broken out. In the store, track read rate and p99 per partition: a hot partition is one key far above the rest. Discord reported p99 history reads going from 40–125 ms on Cassandra to about 15 ms after the move, and that p99 is the number I would watch.

Ask: For every message and every recipient, who may see it? And what happens the instant a moderator takes a role away?

Good answers name: Compute from bitfields on demand, cached per role set and channel, invalidated on change, Precompute and store an ACL per (member, channel), Central permission service (Zanzibar-style), Check only on send, and hide channels in the client.

Our pick: Follow the documented algorithm. The owner gets everything; otherwise base is @everyone’s bits OR every one of the member’s roles, and ADMINISTRATOR means everything. In a channel apply the @everyone overwrite (deny, then allow), then all of the member’s role overwrites together (OR of denies, OR of allows, deny then allow), then the member’s own overwrite. No VIEW_CHANNEL implies no other channel permission, and a timeout leaves only VIEW_CHANNEL and READ_MESSAGE_HISTORY. The guild actor caches results per (role set, channel) plus per member override, so 3,000 role combinations across 500 channels is at most 1.5 M small entries, filled lazily. Checks run at send (SEND_MESSAGES, MENTION_EVERYONE, slowmode bypass), at history read (READ_MESSAGE_HISTORY) and at fan-out (VIEW_CHANNEL per recipient). A change invalidates what it touches: a role’s bits clear every role set containing it, an overwrite clears one channel, a member’s roles clear one member. The guild then tells each affected session which channels appeared or disappeared, in order with the message stream.

1. A moderator removes Alice’s role while a message in a channel it unlocked is being dispatched. Can she see it?
   Both go through the guild actor in one order. If the role change is applied first, the fan-out check reads the new permissions and skips her. If the message was dispatched first, she got it while she could still see the channel, which is correct, and the next event tells her client the channel is gone so it stops showing it. History reads check again, so she cannot scroll back afterwards.
2. Doesn’t asking the guild on every send overload the biggest guilds?
   It is one call per message, and the busiest guild sends maybe a few hundred messages a second; each check is a cached lookup of microseconds. That is tiny next to fan-out, which relays have already taken off the owner. The alternative, the API loading guild state itself, would put a second copy of every guild’s roles in the API tier and a race between the two copies.
3. What is slowmode and where is it enforced?
   A per-channel minimum gap between one member’s messages, from seconds up to six hours in Discord’s API. The guild keeps the last send time per (channel, member) in memory, checks it in the same authorise call and returns a retry-after; members with manage permissions skip it. Losing it on a guild restart lets a few members post early once, which is acceptable. It sits beside the edge’s per-user and per-route limits: those protect the system, slowmode protects the conversation.
4. How do you test that the computation is right?
   Property tests against a reference implementation of the documented algorithm, over generated guilds with random roles and overwrites, covering the edge cases: owner, administrator, timeouts, missing VIEW_CHANNEL, a deny on @everyone with an allow on a role. Then a shadow check in production: sample fan-out decisions, recompute them from scratch and alert on any disagreement. A permission bug is a privacy incident, so it deserves the rigour of money.
5. The owner edits @everyone in a 1 M-member guild. What does it cost?
   Every role set changes, so the cache is cleared and refilled lazily as checks arrive, a few thousand role sets times the channels in use. The expensive part is telling sessions what changed: the guild recomputes visibility once per online role set and sends channel updates only to sessions whose visible channels changed, through the relays. It is a burst of seconds, rate limited per guild, and rare.

Ask: Presence changes outnumber messages. How do you show who is online and what they are playing without presence drowning everything else?

Good answers name: Broadcast every change to every online member of every shared guild, Coalesce per user, deliver only to sessions displaying the member, drop the rest, Fetch presence on demand when a profile or list opens, Aggregate online counts only.

Our pick: The presence service folds a user’s sessions into one status and activity list and emits at most one update per user every few seconds, so a game started and stopped within the window costs nothing. It sends the update straight to friends’ sessions and to each of the user’s guilds. Small guilds forward it to all online members, which is cheap. Large guilds forward it only to sessions whose subscribed member-list ranges contain that member, and above a size threshold drop everything else and show an approximate online count. Bots receive presence only with the privileged GUILD_PRESENCES intent, and big guilds send no offline members at identify (Discord’s large_threshold), both of which exist for exactly this load. Under pressure presence is the first thing shed. With about 50 real viewers per change, deliveries fall from ~170 M to ~1.7 M a second.

1. A user is online on desktop and idle on mobile. What do others see?
   One status: online if any session is active, idle only if all are idle, and an explicitly chosen status such as do not disturb or invisible applies to every session. Activities are merged, so the desktop’s game shows. The aggregation happens once in the presence service, so guilds and friends get one update, not one per device.
2. The presence service crashes.
   Messages keep flowing and statuses freeze. A replacement rebuilds state as gateways re-report each session’s status, which they do on a short interval, so presence is right again within a minute. Nothing durable is lost, because presence is derived from live sessions and was never stored. This is why it is separate from anything that carries messages.
3. How does the member list work with 100,000 online?
   The guild keeps a sorted list of online members per visibility group: hoisted role groups first, then members by name. A client subscribes to the rows it shows, typically 0 to 99 plus wherever it has scrolled, gets a SYNC of those rows and then INSERT, UPDATE and DELETE operations at indexes. A member going idle is one row update for the few hundred sessions showing them. Channels with identical view permissions share one list, so a guild has a handful of lists, not 500.
4. Typing indicators in a channel with 5,000 people looking at it?
   A typer sends at most one typing event every ~8 seconds; it goes only to sessions viewing that channel and is never stored. When more than a few people type at once the client shows "several people are typing", so the guild can send a capped sample instead of every typer. Like presence it is lossy by design and shed early under load.
5. How would you measure presence cost?
   Deliveries per presence change by guild size, the share of updates dropped because nobody displayed the member, and gateway outbound bytes by event type. If presence’s share of bytes creeps up after a client release, the usual cause is a subscription leak: a client keeping ranges subscribed after scrolling away. Updates per user per minute also catch clients reporting activity changes too often.

Ask: Why not carry voice over the same gateway connection, or connect people directly to each other? And which server hosts a call?

Good answers name: Regional SFUs allocated by the guild per voice channel, signalling via the gateway, media over UDP, Peer-to-peer mesh between participants, Mixing server (MCU), Voice over the gateway WebSocket.

Our pick: The guild owns voice state and allocates. On the first join of a voice channel it reads the healthy servers and their load for the region nearest the participants from discovery, picks the least loaded, and gives each joiner the endpoint and a short-lived signed token over the gateway. The client opens a signalling WebSocket to that server and sends encrypted Opus over UDP, only while speaking; Discord’s 2018 post describes Salsa20-based encryption on its native clients, with browsers on standard WebRTC. The server forwards each speaker’s packets to everyone else without decoding them. If it dies it drops out of discovery, clients report the broken socket over the gateway, the guild allocates another server and pushes the new endpoint, and audio is back in a second or two. Discord’s 2018 figures, about 2.6 M concurrent voice users on 850+ servers in 13 regions with over 220 Gbps of egress, are the same order as this design at 3 M.

1. Half the participants are in Europe and half in the US. Which region?
   One server per call keeps it simple, so the region is a compromise: the first joiner’s, or the one that minimises the worst latency once several have joined. Moving a live call is possible (allocate in the new region, push new endpoints) but costs a glitch, so I would do it only when most participants are elsewhere. For a stage with thousands of listeners one server’s capacity is the limit, and listeners would be fed by relay servers that receive the speakers once, which is Zoom’s cascading idea.
2. How many voice servers do you need?
   Bandwidth is not the limit; packets are. A listener receives ~1.5 streams, about 120 kbps but 75 packets a second, so 3 M listeners is ~225 M packets a second. A server forwarding a few hundred thousand packets a second means on the order of a thousand servers, consistent with Discord’s 2018 fleet of 850+ servers carrying 120 M packets a second. Size by packets per core, then add regional headroom so a server’s calls can fail over.
3. Why a signed token instead of the voice server asking the guild?
   So admission needs no call back: the token names the user, guild, channel and server, expires in a minute, and is signed with a key the voice fleet trusts. A server can admit a thousand joins a second without touching the guild tier. Kicking someone is a rare message from the guild telling the server to drop that user.
4. Can you end-to-end encrypt voice through an SFU?
   Yes, because an SFU forwards without decoding: clients encrypt each frame with keys the server never has, and the server routes on the unencrypted packet headers. The hard part is key agreement in a group whose membership changes, which needs a group key protocol so a new joiner cannot decrypt earlier audio. Discord announced end-to-end encryption for audio and video in 2024, its DAVE protocol built on MLS, which works this way.
5. How do you measure voice quality?
   Clients report packet loss, jitter, round-trip time and jitter-buffer underruns per call, tagged by server and region. The p95 of those per server finds a bad host or a congested route within minutes, and the allocator can stop placing calls on servers whose recent quality is poor. On the servers, packets per second per core and NIC drops say when a region needs more capacity.