Design Discord

Community chat and voice: servers with a million members, years of history, roles and presence, for 20 M concurrent users.

Last updated 2026-10-27. Difficulty: hard. Patterns: real-time, fan-out, actors, presence, wide-column. Reported at Discord, Meta, Microsoft, Slack, Roblox, Riot Games.

Functional requirements

• Servers (guilds) with text channels. A guild has channels grouped in categories and can have 1 M+ members. DMs and group DMs are private channels outside any guild and work as in Slack.
• Send and receive messages in real time. Text, mentions, replies, edits and deletes. Sent over REST, received over one gateway WebSocket per client.
• History that scrolls back years. Newest page on open, then pages before a message id; jump to a message link.
• Roles and permissions. Roles carry permission bitfields; channels override them per role or per member. Decides who sees a channel, who can send, who can ping @everyone. Slowmode per channel.
• Presence and typing. Online, idle, do not disturb, offline, plus activities such as "Playing X". Typing shows in the channel you have open.
• Member list. The sidebar of who is online, grouped by role. Must work in a guild with 100 k people online.
• Unread indicators and mention counts. Per user per channel, the same on every device. @everyone and role pings count too.
• Voice channels. Join a channel and talk with everyone in it. Video and screen share ride the same servers and are not designed separately here.
• Bots. Bots connect to the gateway like users. A bot in a million guilds must split its connection into shards.
• Out of scope. Search, push notifications, the file upload pipeline, AutoMod and moderation (see Content Moderation), Nitro and payments, the SFU internals (see Zoom).

Non-functional requirements

• Scale (150 M MAU · 20 M concurrent). Concurrency sizes the gateway fleet; 4 B messages a day size storage. Neither is the hardest number in the design.
• Largest guild (1 M members · 100 k online). The requirement that drives the hardest tradeoff. One process orders a guild’s events, so everything that scales with online sessions has to be moved out of it or cut down to what each client displays.
• Delivery latency (p99 < 500 ms). From send to every session viewing the channel in the region, in a 100 k-online guild too. Sessions not viewing it get badges within seconds rather than every message.
• Ordering (total order per channel). Snowflake ids sort by time; each guild applies its events in one order, so a role change and a message never race.
• Durability (no loss after 200 OK). A message is written at quorum before it is acknowledged or delivered. The live stream may drop an event, because history is the source of truth.
• Availability (99.99 % messaging). Presence, typing and member lists degrade first, on purpose. Voice runs on its own fleet, so a chat incident does not cut calls.
• Voice latency (< 200 ms mouth to ear). Within a region. Drives regional voice servers and media over UDP.
• Permissions (checked at send, read and fan-out). A member who loses a role stops receiving that channel from the next event on. A hidden channel never leaks through history, mentions or the member list.

Back-of-envelope estimates

• Messages per second: ~46 k avg · ~115 k peak. 4 B messages/day ÷ 86 400 s ≈ 46 k/s. Evenings run about 2.5× the average, so ~115 k/s at peak. One partition takes thousands of writes a second, so writes are not the hard part.
• Message deliveries per second: ~4.6 M avg · ~12 M peak. DMs reach one or two sessions and big guilds thousands, but delivering only to sessions that have the guild or the channel open keeps the average near 100 sessions per message: 46 k × 100 = 4.6 M deliveries/s, ~12 M at peak. Each one is filtered by permission first.
• Gateway nodes: ~400. 20 M concurrent sessions at peak. A session holds its guild subscriptions and a replay buffer budgeted at ~100 KB, so 50 k sessions is ~5 GB per node with CPU left for compression and delivery: 20 M ÷ 50 k = 400 nodes.
• Presence deliveries, naive: ~170 M/s. 20 M users each changing status or activity about every 10 minutes is 20 M ÷ 600 = 33 k changes/s. Sending each change to the ~5,000 online members who share a guild with that user: 33 k × 5,000 ≈ 170 M/s, about 35× the message fan-out. Sending only to the ~50 sessions displaying the member brings it to ~1.7 M/s. Presence, not chat, is the biggest stream.
• One event to the biggest guild: ~200 ms of one core. 100 k online sessions × ~2 µs per send = 200 ms for a single event from one process. A big guild produces dozens of events a second (messages, joins, role and voice changes), so one process cannot fan out alone. Relays of up to 15 k sessions turn it into ~7 sends from the guild.
• Message storage per year: ~440 TB. ~300 B per message with ids, author, flags and attachment references (files live in object storage). 4 B × 300 B = 1.2 TB/day ≈ 440 TB/yr before replication, ~1.3 PB with three replicas, about 1.5 trillion messages a year.
• Busiest channel per day: ~520 MB. A channel sustaining 20 messages/s stores 20 × 86 400 × 300 B ≈ 520 MB/day, so a 10-day bucket would be a 5 GB partition. Keeping partitions under ~100 MB means a 1-hour bucket for that channel (~22 MB), while a typical channel fills a 10-day bucket with a few megabytes.
• Voice egress: ~360 Gbps. Assume 3 M people in voice. Opus at ~64 kbps plus packet overhead is ~80 kbps per stream, and a listener receives about 1.5 streams because clients send only while speaking: ~120 kbps and ~75 packets a second each. 3 M × 120 kbps = 360 Gbps, more bandwidth than all of chat.

Components

• Discord client (desktop · web · mobile · bots): Holds one gateway WebSocket for everything pushed to it (messages, presence, member-list ranges, voice state) and uses REST for actions such as send, edit and history. Remembers the last sequence number it saw, so a dropped socket resumes instead of reloading. Sends a nonce with each message so a retried POST cannot post twice. A bot is the same client without a UI, and a big bot opens one connection per shard.
• API edge (REST · auth · rate limits): Terminates TLS, validates the token and applies rate limits before anything else runs: per-route buckets keyed by the route and its major id (channel, guild, webhook), a global limit per user or bot (Discord documents 50 requests a second for bots), and a ban for clients that keep sending invalid requests. Returns 429 with retry_after and bucket headers so good clients pace themselves. Stateless; counters live in a sharded in-memory store.
• API service (send · history · Snowflake ids): The write path for messages and the front door for history. On a send it asks the guild for the member’s permissions and slowmode, assigns a Snowflake id (42 bits of milliseconds, then worker, process and sequence), writes through the data service, returns 200, then hands the event to the guild for live fan-out and to the event log. It never fans out itself, so a message costs it the same in a 10-member guild and a 1 M-member one.
• Data services (Rust · routed by channel): A thin layer between the API and the messages store, one endpoint per query. Requests are routed by consistent hashing on channel_id, so every read of a channel lands on the same instance, and identical reads in flight at the same time are coalesced into one database query whose result goes to every waiter. Discord has written about building exactly this in Rust in front of ScyllaDB to tame hot partitions.
• Messages store (ScyllaDB · (channel, bucket)): Wide-column store holding every message: partition (channel_id, bucket), clustering message_id descending, with the bucket computed from the Snowflake’s timestamp. Buckets default to 10 days, the width Discord described in 2017, and the busiest channels get narrower ones so no partition passes ~100 MB. LOCAL_QUORUM writes; about 440 TB a year before replication.
• Sessions & presence (user → sessions · status): Knows every live session of every user and folds them into one status (online beats idle) plus activities like "Playing X". Coalesces changes so a user broadcasts at most one update every few seconds, sends it to friends’ sessions, and hands it to the user’s guilds, which decide who actually sees it. Also the directory for events aimed at one user, such as a read ack to their other devices.
• Event log (Kafka · key = channel_id): Durable stream of message events for consumers that must not miss one: mention counting in read states, plus search, AutoMod and push, which are out of scope here. Keyed by channel_id so a channel’s events stay in order. Live delivery deliberately does not go through it: a consumer hop adds latency, and a missed live event is recovered from history anyway.
• Read states (user × channel · LRU): One record per user per channel: the last message id read, a mention count and a few flags. Held in memory in an LRU sharded by user_id, with each change committed to a wide-column table about 30 seconds later or on eviction, which is how Discord has described its own read states service (rewritten from Go to Rust to remove GC pauses). Kept apart from messages because it is per user and changes on every read, while a message is written once.
• Gateway (WebSocket · ~50 k sessions/node): Holds the persistent WebSockets. Each session has an id, a sequence number stamped on every event it is sent, and a bounded replay buffer, so a client that drops can RESUME with its last sequence number and get only what it missed. On identify it subscribes the session to each of its guilds using a local copy of the guild ring, tracks which guild and channel the client is looking at, and compresses the stream. Bots connect here too, one session per shard.
• Guild actors (one process per guild): Each guild is one in-memory process that owns its state (channels, roles, overwrites, members, voice states) and its connected sessions, so every event in the guild is applied in one order. It caches each role set’s computed permissions per channel, filters every event by VIEW_CHANNEL, and sends one batch per gateway node rather than one message per session. Placed on guild nodes by a hash ring on guild_id; a dead node’s guilds restart elsewhere from the guilds DB.
• Guilds DB (keyed by guild_id): Durable home of guild state: guilds, channels, roles with their permission bitfields, channel overwrites, members and their role ids. Guild actors load from it on start and write through on every change, so a restarted actor rebuilds exactly. A 1 M-member guild is about a million member rows of ~100 bytes, ~100 MB, which loads in seconds.
• Voice servers (regional SFUs · UDP): Selective forwarding units in many regions: each speaker sends Opus packets once and the server forwards them to everyone else in the voice channel without mixing. One server hosts a channel’s call, chosen by the guild from the least-loaded healthy servers in the region nearest the participants. Signalling is a WebSocket to the server, opened with a token the guild hands out over the gateway; media is UDP and never peer to peer, which keeps users’ IP addresses private.
• Fan-out relays (≤ 15 k sessions each): For guilds too big for one process to fan out, the guild hands its session list to relays, each owning a slice of up to about 15,000 sessions, the figure Discord has published. The guild sends an event once per relay; each relay filters its slice (who is viewing the channel, who may see it, who is passive) and batches by gateway node. Relays also take session joins and leaves, so the guild process stays free for state changes.
• Service discovery (etcd · guild ring · voice load): Small, strongly consistent registry of which nodes are alive. It holds the hash ring that places guild processes on guild nodes, which every gateway and API node keeps a local copy of and watches, and the health and load of every voice server, which the guild reads when it allocates a call. Read from local caches, so it is never on the per-message path.

User flows

1. Send a message in a channel. The core path. The message is durable before the sender sees it posted, and every session that should see it gets it within half a second, with permissions checked twice: once to send, once per recipient.
   1. Client POSTs the message with a nonce; the edge authenticates it and applies rate limits. Limits apply per route and major id (this channel) and per user, before any work is done; slowmode is checked later because it needs guild state. The nonce lets a retry after a timeout return the original message instead of posting twice, and lets the client match the server’s copy to the one it drew as pending.
   2. API service asks the guild actor whether this member may send here and whether slowmode allows it. The guild holds computed permissions in memory, so the check is a lookup: VIEW_CHANNEL and SEND_MESSAGES, MENTION_EVERYONE if the text pings @everyone, and the timeout flag. Slowmode is a last-sent time per (channel, member) in the same process; members with MANAGE_MESSAGES or MANAGE_CHANNELS skip it.
   3. API assigns a Snowflake id and writes the message through the data service. The id’s top 42 bits are milliseconds since Discord’s 2015 epoch, so the bucket comes from the id with no lookup: bucket = timestamp ÷ the channel’s bucket width. The LOCAL_QUORUM write is the durability point; nobody else is shown the message before it. About 300 bytes per message with metadata.
   4. API returns the stored message to the sender. The client swaps its pending copy for the real one, matched by nonce. The sender’s own sessions also receive the MESSAGE_CREATE a moment later and drop it as a duplicate by id.
   5. API dispatches MESSAGE_CREATE to the guild actor, which picks the recipient sessions and sends one batch per gateway node. Recipients are sessions whose member has VIEW_CHANNEL here, read from the permission cache: a second check, because a role change can land between the send and the fan-out. In a small guild every online member’s session gets it; in a large one only sessions viewing the channel do. Batching by node means a guild with 2,000 online spread over 300 nodes costs 300 sends, not 2,000.
   6. Each gateway node stamps the next sequence number for each session, buffers the event and writes it to the socket. Sequence numbers are per session, not per channel: they order everything a session receives (messages, presence, member updates, voice states) and are what RESUME uses. Clients still sort a channel’s messages by id, which fixes the rare pair dispatched out of id order by two API nodes.
   7. API publishes the event to the log; the read-state service raises mention counts for the users it pings. Mention counts must survive a guild actor or a gateway dying mid-fan-out, so they come from the durable log, not the live path. The consumer is idempotent on (user, channel, message id), so a redelivered event cannot count twice. Plain unread needs no write at all: a channel is unread when its last message id is above the user’s last read id.
2. Open a busy channel and scroll back. The main read path. Opening a channel is a REST read of the newest page; the hard case is thousands of people opening the same channel within seconds, which would aim a storm of identical reads at one partition.
   1. Client tells its gateway session which guild and channel it is now looking at. This makes the session active for that channel: in large guilds only sessions viewing a channel get its messages and typing. The gateway forwards it to the guild, or to the relay that owns the session.
   2. Client asks for the newest 50 messages; the API checks VIEW_CHANNEL and READ_MESSAGE_HISTORY with the guild. Reading history is a separate permission from viewing: a channel can show new messages to a member without exposing what was said before. The check is the same cached lookup as on send.
   3. API calls the data service instance that owns this channel on the hash ring. Routing by channel_id is what makes coalescing work: every reader of a channel, from every API node, arrives at the same instance. If an instance dies its channels move to the next one on the ring, losing nothing but in-flight coalescing.
   4. If an identical query is already in flight, the request joins it; otherwise the data service reads the newest bucket. With 30,000 people opening an announcement in a minute, reads arrive at up to ~3,000 a second, but the store sees about one query per in-flight window: at ~5 ms per read, at most ~200 a second however many clients ask. If the newest bucket holds fewer than 50 messages, the service also reads the previous non-empty bucket and merges.
   5. Scrolling back asks for messages before the oldest id on screen; the service walks buckets backwards, skipping empty ones. Any id’s bucket can be computed from the id, so "before X" starts in X’s bucket. A quiet channel can have a hundred empty 10-day buckets in five years; a small per-channel list of non-empty buckets lets the walk jump to the next one with data, which is how Discord described fixing the same problem.
   6. Client acks the newest message it has read; the read-state service moves the marker and clears the mention count. Acks are debounced on the client (on channel switch or after a few idle seconds), and the service updates memory at once and commits to its table 30 seconds later, so a burst of acks becomes one write. Losing a few seconds of acks in a crash only brings back an unread dot.
   7. The ack reaches the user’s other devices so their badges clear too. The sessions directory knows the user’s other live sessions and the gateway nodes they sit on. Read on the laptop, cleared on the phone within a second.
3. An @everyone in a 1 M-member guild. The scale-breaking case: 100,000 members online, one message that pings them all, and 30,000 of them opening the channel within a minute. Every per-session cost is multiplied by 100,000, so the design decides who gets what.
   1. A moderator posts an announcement with @everyone; the API checks MENTION_EVERYONE and stores it as usual. Without MENTION_EVERYONE the message still posts but the ping is ignored, so the permission gates the expensive part, not the message. Storing it costs the same as in any other guild.
   2. The guild actor, which hands fan-out to relays above a size threshold, sends the event once to each of its seven relays. 100,000 sessions at about 2 µs per send would be 200 ms of one core for one event, and a guild this size produces dozens of events a second. Seven relays of up to 15,000 sessions turn that into seven sends, so the one process that orders the guild’s state never does per-session work.
   3. Each relay filters its slice: the full message for sessions viewing the channel, a mention badge for everyone else who can see it. About 90 % of sessions in a big guild are passive, Discord’s own figure: the guild is not open on that device. Passive sessions get a 100-byte mention hint, not the message, and sessions without VIEW_CHANNEL get nothing. Each relay groups by gateway node, so its 15,000 sessions cost at most a few hundred batches.
   4. Mention counts are not written per member: the read-state service records one @everyone mention for the channel. Writing a counter for each of 1 M members would be a million writes for one message. Instead each channel keeps a short list of broad mentions (@everyone, @here, role pings), and a user’s count is their direct mentions plus the broad ones after their read marker that apply to them. MENTION_EVERYONE is rare and rate limited, so the list stays tiny.
   5. 30,000 members open the channel in the next minute; their history reads coalesce at the data service. They all ask for the same newest page, so nearly every request joins a query already in flight. The partition holding today’s bucket sees about a hundred queries a second instead of thousands, and its three replicas are never the bottleneck.
   6. Opening the guild, each client subscribes only to the member-list ranges it is showing. The sidebar shows about 40 rows of a sorted list of 100,000 online members. The client asks for rows 0 to 99 and gets that slice, then insert, update and delete operations on it as members come and go. Offline members are not listed at all in big guilds.
   7. A member’s presence changes; the guild forwards it only to sessions whose subscribed ranges show that member. Of 100,000 online sessions, the ones displaying this member are a few hundred at most; everyone else never hears about it. Above a size threshold the guild sends no presence outside list ranges at all, and the online count in the header is approximate.
4. A dropped socket, then a dead gateway node. Phones lose signal constantly, so reconnecting must be cheap: a resume replays only the missed events. When a whole gateway node dies its sessions are gone, and 50,000 clients must identify again without stampeding the guild tier.
   1. A phone switches networks and its socket closes; the gateway keeps the session and keeps buffering its events. The session stays alive, detached, for a grace period of about two minutes while its replay buffer fills. Presence does not flip to offline during the grace, so friends never see a flap.
   2. The client reconnects to its resume URL and sends RESUME with its session id and last sequence number. A resume costs a few kilobytes and touches no guild: the gateway replays every event after 4182 from its own buffer, in order. A full identify for someone in 100 guilds means 100 guild subscriptions and a READY of hundreds of kilobytes.
   3. If the gap is older than the buffer holds, the gateway refuses the resume and the client identifies from scratch. The buffer is bounded per session (about 2 minutes or 1 MB), so a laptop closed for an hour cannot resume. INVALID_SESSION tells it to identify; history is then read from the store, which is the source of truth, so nothing is lost, only slower.
   4. Later the gateway node itself crashes; guilds and relays see its sessions vanish and drop them. The node’s 50,000 sessions and their buffers lived in its memory and are gone. Guild actors and relays monitor the gateway nodes they deliver to and remove a dead node’s sessions, so fan-out stops sending into it within seconds.
   5. Clients’ resumes fail, so they back off with jitter and identify on other nodes through an admission queue. 50,000 identifies in one second would be millions of guild subscriptions in a burst. Jittered backoff spreads them over tens of seconds, and each node admits identifies at a fixed rate, the same idea as the identify concurrency limits Discord documents for bots.
   6. The new node subscribes the session to each guild via its cached guild ring; a guild with nobody online is loaded from the guilds DB first. Each guild replies with what this session needs: channels, roles and the member’s own data, plus the online member list only for small guilds. Big guilds send no member list until the client subscribes to ranges, which keeps a heavy user’s identify from pulling megabytes. Actors for guilds idle for a while are unloaded and come back from the DB in milliseconds.
   7. The gateway loads the user’s read states and sends READY, then a GUILD_CREATE as each guild answers. READY goes out as soon as the session exists, with guilds marked unavailable, and each guild arrives as its own event, so one slow guild does not hold up the rest. Read states come from the in-memory LRU; a user online minutes ago is almost always still cached.
   8. Presence hears the user is online again but broadcasts nothing, because the status never changed. The presence service kept the user online through the reconnect window, so friends and guilds see no offline and online flap. This is why presence is debounced rather than mirrored from socket events.
5. Join a voice channel and talk. Voice runs on its own fleet: the gateway carries only signalling, and media goes over UDP to a voice server near the participants. Chat and voice fail independently.
   1. Client sends a voice state update over the gateway: join voice channel V. Voice state belongs to the guild like everything else, so joining, muting and moving channels are ordered with the guild’s other events. CONNECT and the channel’s user limit are checked from the permission cache.
   2. If V has no voice server yet, the guild picks the least-loaded healthy one in the nearest region. Voice servers report health and load to discovery; the guild reads the candidates for the region nearest the first participant, or a region pinned on the channel. Everyone in V uses that one server, so servers never forward to each other for an ordinary call. Discord’s 2018 post describes the same allocation through etcd, then per guild rather than per channel.
   3. The guild records the voice state, tells the guild who joined, and sends this session the server’s address and a token. VOICE_STATE_UPDATE goes to every session that can see V, which is what draws you in the channel’s participant list. The token is signed, short-lived and names the user, channel and server, so the voice server can admit the client without calling the guild.
   4. Client opens a signalling WebSocket to the voice server, then starts sending UDP media. Signalling agrees the encryption mode and keys, the client’s SSRC and the UDP address; then audio flows as Opus, ~64 kbps plus headers, about 80 kbps per speaker in 20 ms frames. Clients send only while speaking, so a 10-person channel usually carries one or two streams.
   5. The voice server forwards each speaker’s packets to everyone else in the channel without decoding them. An SFU copies packets and mixes nothing, so each listener controls per-user volume and the server’s cost is packets, not CPU. A listener with two people talking receives about 160 kbps. All media goes through the server rather than peer to peer, which keeps users’ IP addresses private.
   6. The voice server dies; clients notice, the guild allocates a new one and the call carries on. The dead server drops out of discovery when it misses health checks, and clients see their voice socket close and report it over the gateway. The guild picks a new server and sends every participant a fresh VOICE_SERVER_UPDATE, and audio is back in a second or two. Text chat never noticed, because it never touched the voice fleet.

Deep dives

The theory behind it

• Group chat and channel fan-out: Delivering messages to groups of every size: small groups versus huge channels, fan-out on write to inboxes versus reading from a channel log, membership storage, per-member read state and unread counts, read receipts, mentions, large group limits and notification throttling.
• Presence and managing millions of connections: Holding millions of persistent WebSocket connections and showing who is online: connection gateways, routing messages to the right server, heartbeats, presence with TTLs, fan-out of status changes, typing indicators, reconnect storms and deploys.
• Data modelling for Cassandra and DynamoDB: How to design tables for wide-column and key-value stores: query-first modelling, partition keys and sort keys, one table per query versus single-table design, time-bucketed partitions, secondary indexes, avoiding hot and huge partitions, and consistency options.

Related

• Design Slack
• Design a News Feed
• Design a Notification System
• Design Instagram
• All 83 system design interview questions
• Run this question for a friend: the interviewer kit