Design Slack

• Workspaces with public and private channels — Users belong to one or more workspaces; channels are scoped to a workspace.
• Direct messages and group DMs — Model as a channel with a fixed member set so one code path handles both.
• Send and receive messages in real time — Text, mentions, emoji reactions; edits and deletes propagate.
• Message history with pagination — Scroll back through years of history; jump to a date or a permalink.
• Presence and typing indicators — Online / away / offline, plus "X is typing…" in the current channel.
• Unread counts and @mention badges — Per channel per user, consistent across all of a user's devices.
• Threads and reactions — Replies attach to a parent message; reactions are counters keyed by emoji.
• File uploads and previews — Images, PDFs, snippets. Files are attached to messages and searchable by name.
• Full-text search over messages and files — Scoped to channels the searcher can see.
• Push notifications when offline — Mentions and DMs go to mobile push; channel messages honour per-channel preferences.

• Scale (10M DAU · 1M concurrent) — Concurrency is what sizes the real-time tier; DAU sizes storage and history reads.
• Delivery latency (p99 < 500 ms) — Sender to every online recipient within one region. Cross-region can be ~1 s.
• Ordering (total order per channel) — All clients see the same order within a channel. No ordering guarantee across channels.
• Delivery guarantee (at-least-once + dedupe) — A message is never lost once the sender sees an ack. Duplicates are tolerated and deduped client side.
• Durability (no message loss) — History is the product. Acked messages survive node, AZ and region failures.
• Availability (99.99 %) — Roughly 52 minutes of downtime a year. Degraded mode (send works, presence stale) is acceptable.
• Multi-region — Users are worldwide; a workspace is homed in one region but members connect from anywhere.
• Security — Private channel membership enforced on every read and on every fan-out. Encryption in transit and at rest.

• Concurrent connections: ~1 M — 10 M DAU, each online ~2.4 h/day spread over a 24 h day gives 10 M × 2.4 / 24 = 1 M average concurrent. Peak (US morning) is ~2–3× the average, so plan for ~2.5 M connections.
• Messages sent per second: ~5 k avg · 15 k peak — Assume 40 messages per DAU per day. 10 M × 40 = 400 M/day ÷ 86 400 s ≈ 4.6 k/s. Peak 3× ≈ 15 k/s. Writes are not the hard part.
• Fan-out deliveries per second: ~500 k–1 M — Median channel has ~10 members but the distribution is heavy-tailed: a few #general channels have 50 k. Assume an average of 100 online recipients per message: 5 k × 100 = 500 k pushes/s, peaking near 1 M/s. This is the hard part.
• WebSocket servers: ~50–100 — A tuned server holds ~50 k idle connections; budget 25 k per box to leave headroom for fan-out CPU. 2.5 M ÷ 25 k = 100 servers at peak, in a few AZs.
• Storage per year: ~150 TB — Average message ~1 KB after metadata and indexes. 400 M/day × 1 KB × 365 ≈ 146 TB/yr. Replication ×3 → ~450 TB/yr raw. Files are far larger but go to object storage, not the message store.
• Read / write ratio: ~50 : 1 — Every message is read by every member, and history loads re-read old messages. Reads dominate, but most are served from the client cache or a hot recent-messages cache, so the DB read load is a fraction.
• Presence updates per second: ~50 k — 1 M concurrent × heartbeat every 30 s = 33 k/s, plus explicit status changes. Each presence change can fan out to every viewer of that user, so presence is cheap to ingest and expensive to broadcast.

• Web / Mobile client — Holds a WebSocket for real-time events and uses REST for everything else. Keeps a local cache of recent messages per channel and a per-channel cursor (last seen message id) used to resync after reconnect. Generates a client message id for idempotent sends.
• API gateway (REST · auth · rate limit) — Terminates TLS, validates the auth token, enforces per-user rate limits and routes REST calls (history, search, uploads, channel management) to the right service. Stateless, horizontally scaled behind a load balancer.
• WebSocket gateway (sticky, long-lived) — Holds the long-lived connections. On connect it authenticates, registers (user, device) → server in the connection registry and subscribes to the topics for that user's channels. On send it validates, forwards to the Message service and later pushes deliveries to the sockets it owns. ~25 k connections per box.
• Connection registry (Redis cluster) — Maps user_id → set of (ws_server, connection_id) with a TTL refreshed by heartbeats. Lets any service find where a user is connected. Also backs presence (online/away with last-seen timestamps) and pub/sub channels used to route deliveries to the right WebSocket server.
• Channel service — Channels, memberships, permissions. Answers "who is in this channel" and "can this user read it". Membership lists are cached hard because every fan-out needs them.
• Message service — The write path. Assigns the message id, checks membership, persists to the messages store, then publishes a message-created event to Kafka. Also serves history reads with cursor pagination and edits/deletes/reactions.
• Presence service — Ingests heartbeats and status changes, writes to the registry with TTLs and broadcasts presence deltas to interested viewers. Deliberately lossy and rate-limited: presence is a hint, not a fact.
• Event bus (Kafka · keyed by channel_id) — Durable log of message events, partitioned by channel_id so a channel's events stay ordered. Consumers: the fan-out path (WebSocket gateways via the registry), notifications, search indexing, unread counters, analytics. Decouples the write path from every downstream reader.
• Notification service — Consumes message events, decides who should get a push (mentions, DMs, per-channel preferences, and only if no active connection saw the message within a short window), batches and sends via APNs / FCM.
• Search indexer (Elasticsearch / OpenSearch) — Consumes message and file events, indexes text with channel_id and workspace_id as filter fields. Queries always filter by the channels the searcher belongs to, so the ACL check happens inside the index query.
• Messages store (Cassandra / Scylla) — Partition key (channel_id, time_bucket), clustering key message_id descending. A history page is one partition read. Write-optimised LSM storage, linearly scalable, tunable consistency (LOCAL_QUORUM writes).
• Metadata DB (Postgres (sharded by workspace)) — Users, workspaces, channels, memberships, preferences. Relational because it needs joins and constraints; sharded by workspace_id because nothing crosses a workspace. Membership hot-set cached in Redis.
• Object storage + CDN (S3 · CloudFront) — Files are uploaded directly from the client to object storage using a pre-signed URL, so file bytes never flow through application servers. Messages reference the file by id; downloads go through the CDN with signed, expiring URLs.
• Push providers (APNs · FCM) — Apple and Google push services. Best-effort delivery, rate-limited, with device tokens that expire. Treat as unreliable: the client resyncs from the cursor on open regardless of whether the push arrived.

1. Send a message in a channel — The core path. Sender presses enter; every online member in the channel sees the message within 500 ms, and it is durably stored before the sender sees the checkmark.
   1. Client sends the message over its WebSocket with a client-generated id.
   2. WebSocket gateway forwards to the Message service.
   3. Message service checks membership and assigns a message id.
   4. Message is written to the messages store with quorum.
   5. Message service publishes a message-created event to Kafka and acks the sender.
   6. Fan-out: a consumer resolves online members and routes to their WebSocket servers via the registry.
   7. Each WebSocket server pushes the message to the connected recipients.
   8. Asynchronously: search indexing and notifications consume the same event.
2. Client reconnects and syncs missed messages — Phones sleep, laptops close, Wi-Fi drops. Reconnect must be cheap for the server and invisible to the user, with no gaps and no duplicates.
   1. Client opens a new WebSocket and authenticates.
   2. Gateway registers the connection and subscribes to the user's topics.
   3. Client sends its per-channel cursors (last message id seen).
   4. Message service returns messages after each cursor, capped per channel.
   5. Client merges by message id, updates unread counts, and advances cursors.
   6. Presence flips to online and is broadcast to viewers.
3. Presence and typing indicators — Cheap to produce, expensive to broadcast, and nobody needs it to be exact. The design leans into that: TTLs, coalescing, and only telling people who are looking.
   1. Client sends a heartbeat every 30 s over the existing socket.
   2. Gateway forwards to the Presence service, which refreshes the TTL in the registry.
   3. A status change (online → away) is broadcast only to users currently viewing this user.
   4. Typing indicator: client sends "typing in channel C" at most every 3 s.
   5. Gateway publishes directly to the channel's pub/sub topic; other members' gateways deliver it.
4. @mention → push notification to an offline user — Notifications must be timely but never spammy: no push for a message the user already saw on their laptop, and no double-push across devices.
   1. Sender uploads an attachment straight to object storage, then sends the message with an @mention; it is persisted as usual.
   2. Notification service consumes the event from Kafka.
   3. It checks the registry: is the mentioned user online with an active client?
   4. User is offline or did not read it: build the push and send to APNs / FCM.
   5. User taps the push; the client opens the channel and runs the resync flow.
5. Search for a message — Full-text search that respects private channel membership and returns in a few hundred milliseconds across years of history.
   1. Messages and file metadata are indexed asynchronously from Kafka.
   2. User submits a query via REST.
   3. API resolves the channels the user may read.
   4. Search service queries the index with a mandatory channel_id filter.
   5. Results are hydrated and returned; clicking one deep-links into history.

Ask: One message to a 50 000-member channel: how does it reach everyone without melting the write path?

Good answers name: Push per recipient, Per-channel subscription (topic per channel), Pull on open (no push for big channels).

Our pick: Hybrid keyed on channel size. Below a threshold (say 500 members) do per-recipient push via the registry: it is exact and simple. Above it, gateways subscribe to a per-channel pub/sub topic when any of their users is a member, and one publish reaches every server that needs it; the server then fans out locally to its own sockets. Offline members are never pushed to at all; they resync from cursors. This keeps per-message cost proportional to the number of gateway servers, which is bounded, instead of the number of members, which is not.

1. A user is a member of 2,000 channels. What happens when they connect?
   Their gateway must subscribe to up to 2,000 topics, which is a burst of subscribe calls on every reconnect. Mitigations: subscribe lazily to channels the client has open or recently viewed and rely on cursor resync for the rest; batch subscribe calls; or subscribe per user and let the fan-out consumer resolve user-to-topic server side for small channels. Slack itself only pushes to channels the client has 'opened' and lets unread counts come from a separate counter service.
2. Where exactly does the per-message work happen for a 50k channel, and what is its cost?
   The fan-out consumer reads one Kafka event and publishes once to the channel topic in Redis pub/sub. Redis then delivers to every subscribed gateway server, maybe 100 of them. Each gateway loops over its local sockets for that channel. So the cost is one publish plus roughly 100 network deliveries plus 50k local socket writes spread across 100 boxes, about 500 writes per box. Nothing does 50k of anything in one place.
3. What if Redis pub/sub loses a message? It is fire and forget.
   It is, and that is acceptable because the durable copy is in Cassandra and Kafka. A client that misses a push notices via the next message's id gap or on the next reconnect and resyncs from its cursor. The realtime path is optimised for latency, the durable path for completeness. If the interviewer wants tighter guarantees, the gateway can periodically reconcile each open channel's latest id against the store.
4. How does the gateway know when to unsubscribe from a channel topic?
   Reference count per (server, channel): increment when a connected user with that channel opens it, decrement on close or disconnect, unsubscribe at zero. Add a grace period so a flapping client does not churn subscriptions. On gateway crash the subscriptions die with the Redis connection, so no leak.
5. How would this change at 10x scale, 10M concurrent?
   Gateway count goes to roughly 1,000, so a single Redis pub/sub node fanning to 1,000 subscribers per channel becomes the hotspot. Shard pub/sub by channel id across a Redis cluster or replace it with a dedicated fan-out tier, a layer of relay nodes each responsible for a slice of gateways. The shape stays the same, you add one more level to the tree.

Ask: Why hold a million persistent connections instead of letting clients poll?

Good answers name: WebSocket, Server-Sent Events + REST for sends, Long polling.

Our pick: WebSocket for the real-time channel, REST for everything that is not latency-sensitive (history, search, uploads, settings). The gateway is deliberately thin so that its only scaling dimension is connections. The registry makes the statefulness manageable: any service can find a user's server, and a gateway crash simply drops its connections, which reconnect elsewhere and resync from cursors within seconds. Keep long polling as a hidden fallback for networks that block upgrades.

1. A WebSocket gateway server crashes with 25k connections. Walk me through the next 30 seconds.
   All 25k clients see the socket close and reconnect with jittered exponential backoff over a few seconds, landing on other gateways via the load balancer. Each registers in the registry, resubscribes and sends its per-channel cursors to fetch anything missed. The registry entries for the dead server expire by TTL, so fan-out stops routing there within a minute; any deliveries routed there in the gap are recovered by the cursor resync. Users see a brief 'reconnecting' state and nothing lost.
2. How do you deploy a new gateway version without disconnecting a million users at once?
   Rolling restart with connection draining: mark the instance unhealthy so the LB stops sending new connections, then send a 'reconnect soon' control frame to existing clients spread over a window of several minutes so they migrate gradually. Only after the window force-close the remainder. Never restart more than a small percentage of the fleet at once.
3. Why is the gateway thin? What would go wrong if it also validated membership and wrote to the DB?
   Because its scaling dimension is connections, which is memory-bound and long-lived, while business logic scales on CPU and request rate. Mixing them means a burst of messages can starve connection handling, and every deploy of business logic disconnects users. Keeping the gateway to auth, framing and routing lets you deploy the Message service many times a day without touching connections.
4. How do load balancers handle a million long-lived connections?
   Layer 4 balancing is enough since there is no per-request routing after the upgrade. The LB needs high connection-table limits, long idle timeouts and no per-request cost. Client-side: after auth, hand the client a specific gateway address or use a consistent-hash LB so that reconnects tend to land on the same server, which keeps subscription churn low. Cloud LBs need explicit idle-timeout configuration or they will drop sockets at 60 seconds; heartbeats every 30 seconds keep them alive.

Ask: How do all clients agree on the order of messages in a channel, without a single global sequencer?

Good answers name: Time-ordered 64-bit id (Snowflake), Per-channel sequence number, Database timestamp / auto-increment.

Our pick: Snowflake-style ids: 41 bits of milliseconds, 10 bits of node id, 12 bits of per-node sequence. Uniqueness without coordination, time order good enough for humans, and a natural cursor. Within a channel, Kafka partitioning by channel_id then gives a single consumer order for fan-out, so all online clients receive the same sequence. Clients sort by id, which resolves the rare case where two messages are delivered out of order. If the interviewer pushes on strictness, describe the per-channel sequence as an upgrade for specific channels rather than the default.

1. Two users send at the same millisecond from different regions. Who wins and does it matter?
   The Snowflake id breaks the tie by node id and sequence, so there is a deterministic order but it is arbitrary with respect to real time. For chat that is fine: both users perceive their own message as sent first and there is no causal dependency between them. If the interviewer pushes, causal order is what matters: a reply must follow the message it replies to, and that is guaranteed because the reply is created after the original was received and acked.
2. Clock skew: a Message service node's clock is 5 seconds ahead. What breaks?
   Its ids sort 5 seconds into the future, so its messages appear after messages sent slightly later by other nodes. Worse, cursors: a client that has seen the skewed id will miss messages from correct nodes with smaller ids on resync. Mitigations: NTP with tight bounds, refuse to start if skew exceeds a threshold, never let the sequence go backwards on the node, and have resync fetch by id but also overlap by a few seconds of time to catch skew.
3. How do you paginate history with these ids? What about jumping to a specific date?
   Cursor pagination: 'give me 50 messages in channel C with id less than X'. Because ids are time ordered and the clustering key is id descending, that is a single range scan. Jumping to a date is the same query: construct the minimal Snowflake id for that timestamp, which is just the timestamp bits with zeros, and query from there. No offset pagination anywhere.
4. If ordering per channel is the requirement, why route through Kafka at all? Cassandra already stored it.
   Kafka gives every downstream consumer the same ordered stream without each of them polling the DB, and partitioning by channel id means the fan-out consumer processes a channel's events in order on one thread. It also decouples write latency from the number of consumers: adding search or analytics does not add latency to send. The order in Kafka matches id order in practice because a single channel's writes are serialised by the partition.

Ask: Which database holds years of messages, and what is the partition key so history reads stay fast as channels age?

Good answers name: Cassandra / Scylla, partition (channel_id, time_bucket), Sharded Postgres / MySQL by channel_id, Single relational database.

Our pick: Cassandra-style wide-column store with partition key (channel_id, time_bucket) and clustering key message_id DESC. Bucket size is chosen per channel volume so a partition stays under ~100 MB. Reads for "latest page" hit the current bucket and, if short, the previous one. Writes use LOCAL_QUORUM for durability within the home region, with async cross-region replication. Metadata (users, channels, memberships) stays relational in Postgres because it needs constraints and joins and is orders of magnitude smaller. Mention that Slack chose sharded MySQL and explain why both are defensible: the interviewer wants to hear you match the store to the access pattern, not name a brand.

1. A partition for (#general, today) is getting hot: 10k writes/sec into one partition. What do you do?
   Shrink the bucket for that channel to an hour, or add a small hash suffix to the partition key for hot channels so writes spread across N partitions, reading N partitions in parallel and merging by id. Choose bucket size per channel from observed volume and store it in channel metadata. Also confirm the real bottleneck: 10k writes/sec to one Cassandra partition is survivable briefly but hurts compaction.
2. How do you handle message edits and deletes in an append-only, LSM-based store?
   Edits are an upsert on the same primary key, which LSM handles as a new version. Deletes write a tombstone. The problem is tombstone accumulation making reads slow in channels with heavy deletion; mitigations are soft deletes (a deleted flag) that are filtered at read time and purged by a periodic job, and tuning gc_grace_seconds. Edit history, if needed, is a separate table keyed by message id.
3. Why not put messages in Postgres like the metadata? One database is simpler.
   Volume and access pattern. Messages are hundreds of TB, append-only and read by range on one key; metadata is gigabytes, relational and needs constraints and joins. A B-tree with billions of rows in one table needs manual sharding and its write amplification on a busy channel index is poor compared to LSM. Slack does run sharded MySQL and it works, but they invested heavily in Vitess; for a fresh design the wide-column store matches the pattern with less operational invention.
4. Multi-region: a workspace homed in the US has members in Europe. Where are writes and reads served?
   Writes go to the home region for the channel to preserve a single order and LOCAL_QUORUM durability. Reads of recent history can be served from an async replica in the reader's region with a few hundred ms lag, which is acceptable for scrollback. Real-time delivery crosses regions via Kafka mirroring or the gateway in Europe subscribing to the US pub/sub tier. The cross-region latency shows up as roughly 100 ms extra on delivery, still under the 1 s cross-region target.
5. What is the retention story? Some workspaces want 90 days, some want forever.
   Cassandra TTL per row is the simple mechanism but cannot be changed retroactively for existing rows. Better: store retention policy on the workspace and run a deletion job per time bucket, dropping whole partitions, which is cheap. Legal hold overrides deletion. Search index deletions must follow the same policy or the index becomes the leak.

Ask: What does "never lose a message" actually mean, and where can duplicates come from?

Good answers name: At-least-once + idempotency at each hop, At-most-once (fire and forget), Transactional exactly-once end to end.

Our pick: At-least-once everywhere, idempotent everywhere. Client sends carry a client_msg_id; the Message service keeps (user_id, client_msg_id) → message_id in Redis for a few minutes and returns the existing id on a retry. The DB write is the durability point; the Kafka publish is made reliable with a transactional outbox (or CDC from the messages table) so a crash between DB write and publish still emits the event. Downstream, every consumer keys on message_id, and clients dedupe on it. Reconnect resyncs from per-channel cursors, which covers any delivery lost while a gateway was dying. Typing and presence are explicitly at-most-once because losing them costs nothing.

1. The client sends, the server writes to Cassandra, then the server crashes before acking. What does the user see and what happens next?
   The client's send times out and it retries with the same client message id. The dedupe entry in Redis (or the DB) returns the existing message id, so no duplicate is written, and the ack arrives. Meanwhile the outbox or CDC has already emitted the event, so recipients got the message once. The user sees a spinner for a few seconds and then a checkmark.
2. Your dedupe store is Redis with a TTL. What if the retry comes after the TTL?
   Then a duplicate can be written. Choose the TTL longer than any plausible client retry window, say 10 minutes versus a client that gives up after 1. For belt and braces, make the dedupe key part of the Cassandra row via a lightweight transaction or a secondary table keyed by (user, client id) with a longer TTL. At Slack's scale the Redis approach with a generous TTL is what is actually used.
3. How does the receiving client know it missed a message rather than there simply being none?
   It cannot from a single message, since ids are not gapless. It knows on reconnect by comparing cursors, and it can periodically ask 'latest id in channel C' for open channels and compare to what it has. Read receipts and unread counts come from a server-side counter, so the badge is correct even when the realtime stream dropped something.
4. Is there any part of this system where at-most-once is the right choice?
   Typing indicators, presence heartbeats and ephemeral cursors. They are superseded within seconds and losing one has no user-visible consequence. Making them durable would add load to Kafka and the DB for zero benefit. Being explicit about which signals are lossy is a mark of a mature design.

Ask: Why is presence the feature most likely to take down a chat system, and how do you make it safe?

Good answers name: TTL keys + subscribe-to-visible + coalescing, Broadcast every change to the whole workspace, Poll presence on demand.

Our pick: Heartbeats refresh a Redis key with a 60 s TTL; expiry means offline, so no cleanup path is needed for crashes. Clients subscribe to presence only for user ids currently visible (DM sidebar, current channel's member list), and the Presence service publishes deltas to those subscriptions through the same pub/sub used for messages. Changes per user are rate limited (one per few seconds) and coalesced, and last-write-wins resolves races between devices. Presence is explicitly at-most-once and served stale under load: when the cluster is stressed, presence degrades first, by design, and messaging keeps working.

1. A user has 5 devices. What is their presence?
   Presence is per user, computed as the max over devices: online if any device is active, away if all are idle, offline if all keys expired. Each device refreshes its own key (user, device) and a small aggregation on read or on change computes the user-level status. Last-write-wins per device avoids races between them.
2. The Redis cluster holding presence dies. What breaks?
   Presence and typing degrade: everyone looks offline or stale until the cluster is back and heartbeats repopulate it, which takes one heartbeat interval. Messaging continues because the connection registry for fan-out should be a separate keyspace with replication, or at least the pub/sub path should survive. This is the argument for keeping presence in its own cluster: it is the component you are most willing to lose.
3. How does the client decide which users to subscribe to for presence?
   Whatever is on screen: the DM sidebar, the member list of the open channel, and the authors of visible messages. Subscriptions are diffed as the UI changes, with a small delay to avoid churn while scrolling. Cap the subscription set at a few hundred; beyond that, show presence only on hover via a point query.
4. What does a 'presence storm' look like and how do you prevent it?
   An office loses Wi-Fi for 30 seconds: 5,000 users go offline then online within seconds, each change fanning to hundreds of viewers, millions of events. Prevention: debounce status changes, do not broadcast offline until the TTL actually expires rather than on socket close, coalesce multiple changes per user per few seconds, and rate limit presence deliveries per receiving connection. Users will not notice a 10 second delay in a green dot.