SysDesignPrep.com
System design interview question

Design Slack

Real-time team messaging: channels, DMs, presence, history and search for millions of concurrent users.

Last updated 2026-09-22. Difficulty: hard. Patterns: real-time, websockets, fan-out, chat. Reported at Anthropic and 7 more with Pro.

Walk through a strong candidate's answer, turn by turn.

The interviewer asks, the candidate answers and draws, and you press Next. Pause to answer yourself at the key decisions, and ask the coach anything along the way.

Walk through itTake the mock interview

Functional requirements

  • Workspaces with public and private channels. Users belong to one or more workspaces; channels are scoped to a workspace.
  • Direct messages and group DMs. Model as a channel with a fixed member set so one code path handles both.
  • Send and receive messages in real time. Text, mentions, emoji reactions; edits and deletes propagate.
  • Message history with pagination. Scroll back through years of history; jump to a date or a permalink.
  • Presence and typing indicators. Online / away / offline, plus "X is typing…" in the current channel.
  • Unread counts and @mention badges. Per channel per user, consistent across all of a user's devices.
  • Threads and reactions. Replies attach to a parent message; reactions are counters keyed by emoji.
  • File uploads and previews. Images, PDFs, snippets. Files are attached to messages and searchable by name.
  • Full-text search over messages and files. Scoped to channels the searcher can see.
  • Push notifications when offline. Mentions and DMs go to mobile push; channel messages honour per-channel preferences.

Non-functional requirements

  • Scale (10M DAU · 1M concurrent). Concurrency is what sizes the real-time tier; DAU sizes storage and history reads.
  • Delivery latency (p99 < 500 ms). Sender to every online recipient within one region. Cross-region can be ~1 s.
  • Ordering (total order per channel). All clients see the same order within a channel. No ordering guarantee across channels.
  • Delivery guarantee (at-least-once + dedupe). A message is never lost once the sender sees an ack. Duplicates are tolerated and deduped client side.
  • Durability (no message loss). History is the product. Acked messages survive node, AZ and region failures.
  • Availability (99.99 %). Roughly 52 minutes of downtime a year. Degraded mode (send works, presence stale) is acceptable.
  • Multi-region. Users are worldwide; a workspace is homed in one region but members connect from anywhere.
  • Security. Private channel membership enforced on every read and on every fan-out. Encryption in transit and at rest.

Back-of-envelope estimates

  • Concurrent connections: ~1 M. 10 M DAU, each online ~2.4 h/day spread over a 24 h day gives 10 M × 2.4 / 24 = 1 M average concurrent. Peak (US morning) is ~2–3× the average, so plan for ~2.5 M connections.
  • Messages sent per second: ~5 k avg · 15 k peak. Assume 40 messages per DAU per day. 10 M × 40 = 400 M/day ÷ 86 400 s ≈ 4.6 k/s. Peak 3× ≈ 15 k/s. Writes are not the hard part.
  • Fan-out deliveries per second: ~500 k–1 M. Median channel has ~10 members but the distribution is heavy-tailed: a few #general channels have 50 k. Assume an average of 100 online recipients per message: 5 k × 100 = 500 k pushes/s, peaking near 1 M/s. This is the hard part.
  • WebSocket servers: ~50–100. A tuned server holds ~50 k idle connections; budget 25 k per box to leave headroom for fan-out CPU. 2.5 M ÷ 25 k = 100 servers at peak, in a few AZs.
  • Storage per year: ~150 TB. Average message ~1 KB after metadata and indexes. 400 M/day × 1 KB × 365 ≈ 146 TB/yr. Replication ×3 → ~450 TB/yr raw. Files are far larger but go to object storage, not the message store.
  • Read / write ratio: ~50 : 1. Every message is read by every member, and history loads re-read old messages. Reads dominate, but most are served from the client cache or a hot recent-messages cache, so the DB read load is a fraction.
  • Presence updates per second: ~50 k. 1 M concurrent × heartbeat every 30 s = 33 k/s, plus explicit status changes. Each presence change can fan out to every viewer of that user, so presence is cheap to ingest and expensive to broadcast.

Components

  • Web / Mobile client: Holds a WebSocket for real-time events and uses REST for everything else. Keeps a local cache of recent messages per channel and a per-channel cursor (last seen message id) used to resync after reconnect. Generates a client message id for idempotent sends.
  • API gateway (REST · auth · rate limit): Terminates TLS, validates the auth token, enforces per-user rate limits and routes REST calls (history, search, uploads, channel management) to the right service. Stateless, horizontally scaled behind a load balancer.
  • WebSocket gateway (sticky, long-lived): Holds the long-lived connections. On connect it authenticates, registers (user, device) → server in the connection registry and subscribes to the topics for that user's channels. On send it validates, forwards to the Message service and later pushes deliveries to the sockets it owns. ~25 k connections per box.
  • Connection registry (Redis cluster): Maps user_id → set of (ws_server, connection_id) with a TTL refreshed by heartbeats. Lets any service find where a user is connected. Also backs presence (online/away with last-seen timestamps) and pub/sub channels used to route deliveries to the right WebSocket server.
  • Channel service: Channels, memberships, permissions. Answers "who is in this channel" and "can this user read it". Membership lists are cached hard because every fan-out needs them.
  • Message service: The write path. Assigns the message id, checks membership, persists to the messages store, then publishes a message-created event to Kafka. Also serves history reads with cursor pagination and edits/deletes/reactions.
  • Presence service: Ingests heartbeats and status changes, writes to the registry with TTLs and broadcasts presence deltas to interested viewers. Deliberately lossy and rate-limited: presence is a hint, not a fact.
  • Event bus (Kafka · keyed by channel_id): Durable log of message events, partitioned by channel_id so a channel's events stay ordered. Consumers: the fan-out path (WebSocket gateways via the registry), notifications, search indexing, unread counters, analytics. Decouples the write path from every downstream reader.
  • Notification service: Consumes message events, decides who should get a push (mentions, DMs, per-channel preferences, and only if no active connection saw the message within a short window), batches and sends via APNs / FCM.
  • Search indexer (Elasticsearch / OpenSearch): Consumes message and file events, indexes text with channel_id and workspace_id as filter fields. Queries always filter by the channels the searcher belongs to, so the ACL check happens inside the index query.
  • Messages store (Cassandra / Scylla): Partition key (channel_id, time_bucket), clustering key message_id descending. A history page is one partition read. Write-optimised LSM storage, linearly scalable, tunable consistency (LOCAL_QUORUM writes).
  • Metadata DB (Postgres (sharded by workspace)): Users, workspaces, channels, memberships, preferences. Relational because it needs joins and constraints; sharded by workspace_id because nothing crosses a workspace. Membership hot-set cached in Redis.
  • Object storage + CDN (S3 · CloudFront): Files are uploaded directly from the client to object storage using a pre-signed URL, so file bytes never flow through application servers. Messages reference the file by id; downloads go through the CDN with signed, expiring URLs.
  • Push providers (APNs · FCM): Apple and Google push services. Best-effort delivery, rate-limited, with device tokens that expire. Treat as unreliable: the client resyncs from the cursor on open regardless of whether the push arrived.

User flows

  1. Send a message in a channel. The core path. Sender presses enter; every online member in the channel sees the message within 500 ms, and it is durably stored before the sender sees the checkmark.
    1. Client sends the message over its WebSocket with a client-generated id. The client id (a UUID) makes the send idempotent: if the socket drops before the ack, the client resends with the same id and the server dedupes. The client renders the message optimistically in a "sending" state.
    2. WebSocket gateway forwards to the Message service. The gateway does no business logic beyond auth and basic size/rate checks. Keeping it thin means it can be scaled purely on connection count.
    3. Message service checks membership and assigns a message id. Membership comes from a Redis-cached set owned by the Channel service. The id is a time-ordered 64-bit id (Snowflake style) so ids sort by time and are unique without coordination.
    4. Message is written to the messages store with quorum. LOCAL_QUORUM write to Cassandra, partition (channel_id, day). Only after this returns is the message considered accepted. This is the durability point.
    5. Message service publishes a message-created event to Kafka and acks the sender. Publish is keyed by channel_id so the partition preserves channel order. The ack carries the server message id and timestamp; the client swaps its optimistic message for the real one. If the Kafka publish fails after the DB write, an outbox table or a CDC stream from the DB guarantees the event is still emitted.
    6. Fan-out: a consumer resolves online members and routes to their WebSocket servers via the registry. For a normal channel: look up members (cached), intersect with the registry to find who is online and on which server, then publish the message to each server's Redis pub/sub channel. For huge channels the gateways subscribe to a per-channel topic instead; see the fan-out deep dive.
    7. Each WebSocket server pushes the message to the connected recipients. Recipients dedupe on message id and insert in id order. Their per-channel cursor advances only when the message is rendered, so a crash mid-delivery is recovered by the resync flow.
    8. Asynchronously: search indexing and notifications consume the same event. These consumers are in separate consumer groups with their own lag budgets. Search can lag seconds; nobody notices. Notifications wait a few seconds to see if the user read the message on an active device first.
  2. Client reconnects and syncs missed messages. Phones sleep, laptops close, Wi-Fi drops. Reconnect must be cheap for the server and invisible to the user, with no gaps and no duplicates.
    1. Client opens a new WebSocket and authenticates. Reconnect uses exponential backoff with jitter so a network blip in an office does not create a thundering herd against one gateway.
    2. Gateway registers the connection and subscribes to the user's topics. Registry entry: user_id → {server, conn_id, ts} with a TTL. Subscriptions are set up before the resync so nothing that arrives during the resync is missed; duplicates are handled by id.
    3. Client sends its per-channel cursors (last message id seen). Cursors are tiny: one 64-bit id per channel. Because ids are time-ordered, "everything after id X" is a single range query per channel.
    4. Message service returns messages after each cursor, capped per channel. If a channel has more than N (say 200) missed messages, return the newest N plus a "gap" marker and let the client page back on demand. This bounds the cost of a user returning from vacation.
    5. Client merges by message id, updates unread counts, and advances cursors. Merge is a set union on message id, so any message that arrived over the new socket during the resync is not duplicated. Unread counts are recomputed from the last-read marker rather than incremented, so they cannot drift.
    6. Presence flips to online and is broadcast to viewers. Presence is the last thing to happen because it is the least important. If it fails, the user can still chat.
  3. Presence and typing indicators. Cheap to produce, expensive to broadcast, and nobody needs it to be exact. The design leans into that: TTLs, coalescing, and only telling people who are looking.
    1. Client sends a heartbeat every 30 s over the existing socket. No new connection, one tiny frame. The gateway also treats any real message as an implicit heartbeat.
    2. Gateway forwards to the Presence service, which refreshes the TTL in the registry. SET user:{id}:presence online EX 60. If the key expires the user is offline. No explicit "offline" message is required, which makes crashes and dropped sockets safe.
    3. A status change (online → away) is broadcast only to users currently viewing this user. Clients subscribe to presence for the people visible on screen (open DM list, current channel members), not for the whole workspace. A 50 k-person workspace would otherwise generate 50 k × 50 k presence traffic.
    4. Typing indicator: client sends "typing in channel C" at most every 3 s. Client-side throttled. The event is ephemeral: it is never persisted and never goes through Kafka.
    5. Gateway publishes directly to the channel's pub/sub topic; other members' gateways deliver it. Fire and forget. Receivers show the indicator and clear it after 5 s if no refresh arrives. Losing a typing event costs nothing, so the whole path skips durability.
  4. @mention → push notification to an offline user. Notifications must be timely but never spammy: no push for a message the user already saw on their laptop, and no double-push across devices.
    1. Sender uploads an attachment straight to object storage, then sends the message with an @mention; it is persisted as usual. The file goes to object storage via a pre-signed URL so its bytes never pass through the gateway or Message service; the message carries only the file id. Mentions are parsed server side into a structured list of user ids attached to the message event, so consumers do not re-parse text.
    2. Notification service consumes the event from Kafka. Its own consumer group, so its lag never slows down real-time delivery. It filters to messages that might warrant a push: DMs, mentions, keywords, channels with "all messages" preference.
    3. It checks the registry: is the mentioned user online with an active client? If online, it schedules a delayed check (e.g. 10 s). If a read receipt for that message arrives from any device in the window, the push is cancelled. This is why you do not get buzzed for a message you are already reading.
    4. User is offline or did not read it: build the push and send to APNs / FCM. Payload contains channel and message ids, not the full text for E2E-sensitive workspaces. Per-user push rate limits and quiet hours are applied here. Multiple mentions in a short window are coalesced into one "3 new mentions in #eng".
    5. User taps the push; the client opens the channel and runs the resync flow. The push is only a wake-up signal. Everything the client renders comes from the resync, so a lost push or a stale payload cannot show wrong content.
  5. Search for a message. Full-text search that respects private channel membership and returns in a few hundred milliseconds across years of history.
    1. Messages and file metadata are indexed asynchronously from Kafka. Index document: {message_id, workspace_id, channel_id, author, text, ts, has_file}. Indexed a few seconds after send. Edits and deletes are applied as updates keyed by message_id.
    2. User submits a query via REST. Query parsing (from:@alice in:#eng before:2025-01) happens at the API layer and becomes structured filters.
    3. API resolves the channels the user may read. One cached call returning the user's channel ids plus all public channel ids in the workspace. This list is the ACL.
    4. Search service queries the index with a mandatory channel_id filter. Index is sharded by workspace_id so a query hits one shard group. Results are ranked by relevance and recency. Because the ACL is a filter inside the query, there is no post-filtering that could leak counts or blow up pagination.
    5. Results are hydrated and returned; clicking one deep-links into history. The hit contains enough to render a snippet. Clicking it loads history around that message id from the messages store, which is a cheap range query thanks to time-ordered ids.

Deep dives

Fan-out strategy for large channels

One message to a 50 000-member channel: how does it reach everyone without melting the write path?

Writes are ~5 k/s but deliveries are ~500 k–1 M/s. The whole difficulty of chat is this multiplication. A design that looks up every member and pushes to each one individually is fine for a 10-person DM and catastrophic for #general in a large company: one send becomes 50 000 registry lookups and 50 000 pushes on the critical path.

Two observations shape the answer. First, only online members need to be pushed to; offline members will resync from their cursor. Second, connections are concentrated on a few hundred gateway servers, so there are far fewer servers than recipients. Delivery should be routed per server, not per user.

  • Push per recipient situational: channels below a few hundred members
  • Per-channel subscription (topic per channel) chosen: large channels; in practice used for all channels above a size threshold
  • Pull on open (no push for big channels) situational: announcement-only channels or very cold channels

The answer: Hybrid keyed on channel size. Below a threshold (say 500 members) do per-recipient push via the registry: it is exact and simple. Above it, gateways subscribe to a per-channel pub/sub topic when any of their users is a member, and one publish reaches every server that needs it; the server then fans out locally to its own sockets. Offline members are never pushed to at all; they resync from cursors. This keeps per-message cost proportional to the number of gateway servers, which is bounded, instead of the number of members, which is not.

A user is a member of 2,000 channels. What happens when they connect?

Their gateway must subscribe to up to 2,000 topics, which is a burst of subscribe calls on every reconnect. Mitigations: subscribe lazily to channels the client has open or recently viewed and rely on cursor resync for the rest; batch subscribe calls; or subscribe per user and let the fan-out consumer resolve user-to-topic server side for small channels. Slack itself only pushes to channels the client has 'opened' and lets unread counts come from a separate counter service.

Where exactly does the per-message work happen for a 50k channel, and what is its cost?

The fan-out consumer reads one Kafka event and publishes once to the channel topic in Redis pub/sub. Redis then delivers to every subscribed gateway server, maybe 100 of them. Each gateway loops over its local sockets for that channel. So the cost is one publish plus roughly 100 network deliveries plus 50k local socket writes spread across 100 boxes, about 500 writes per box. Nothing does 50k of anything in one place.

What if Redis pub/sub loses a message? It is fire and forget.

It is, and that is acceptable because the durable copy is in Cassandra and Kafka. A client that misses a push notices via the next message's id gap or on the next reconnect and resyncs from its cursor. The realtime path is optimised for latency, the durable path for completeness. If the interviewer wants tighter guarantees, the gateway can periodically reconcile each open channel's latest id against the store.

How does the gateway know when to unsubscribe from a channel topic?

Reference count per (server, channel): increment when a connected user with that channel opens it, decrement on close or disconnect, unsubscribe at zero. Add a grace period so a flapping client does not churn subscriptions. On gateway crash the subscriptions die with the Redis connection, so no leak.

How would this change at 10x scale, 10M concurrent?

Gateway count goes to roughly 1,000, so a single Redis pub/sub node fanning to 1,000 subscribers per channel becomes the hotspot. Shard pub/sub by channel id across a Redis cluster or replace it with a dedicated fan-out tier, a layer of relay nodes each responsible for a slice of gateways. The shape stays the same, you add one more level to the tree.

Real-time transport: WebSocket vs long polling vs SSE

Why hold a million persistent connections instead of letting clients poll?

Chat needs the server to push to the client with sub-second latency, and the client to send with the same latency. Polling every second from 1 M clients is 1 M requests/s of mostly empty responses; that is more load than the actual messages. The real choice is between the persistent-connection options.

Persistent connections bring an operational problem: connection state lives on a specific server, so that server becomes stateful. The design handles this with the connection registry and by making reconnect cheap (see the sync flow), rather than by trying to make connections migratable.

  • WebSocket chosen
  • Server-Sent Events + REST for sends situational: read-heavy feeds (notifications, dashboards) where clients rarely send
  • Long polling rejected: fallback only for hostile networks

The answer: WebSocket for the real-time channel, REST for everything that is not latency-sensitive (history, search, uploads, settings). The gateway is deliberately thin so that its only scaling dimension is connections. The registry makes the statefulness manageable: any service can find a user's server, and a gateway crash simply drops its connections, which reconnect elsewhere and resync from cursors within seconds. Keep long polling as a hidden fallback for networks that block upgrades.

A WebSocket gateway server crashes with 25k connections. Walk me through the next 30 seconds.

All 25k clients see the socket close and reconnect with jittered exponential backoff over a few seconds, landing on other gateways via the load balancer. Each registers in the registry, resubscribes and sends its per-channel cursors to fetch anything missed. The registry entries for the dead server expire by TTL, so fan-out stops routing there within a minute; any deliveries routed there in the gap are recovered by the cursor resync. Users see a brief 'reconnecting' state and nothing lost.

How do you deploy a new gateway version without disconnecting a million users at once?

Rolling restart with connection draining: mark the instance unhealthy so the LB stops sending new connections, then send a 'reconnect soon' control frame to existing clients spread over a window of several minutes so they migrate gradually. Only after the window force-close the remainder. Never restart more than a small percentage of the fleet at once.

Why is the gateway thin? What would go wrong if it also validated membership and wrote to the DB?

Because its scaling dimension is connections, which is memory-bound and long-lived, while business logic scales on CPU and request rate. Mixing them means a burst of messages can starve connection handling, and every deploy of business logic disconnects users. Keeping the gateway to auth, framing and routing lets you deploy the Message service many times a day without touching connections.

How do load balancers handle a million long-lived connections?

Layer 4 balancing is enough since there is no per-request routing after the upgrade. The LB needs high connection-table limits, long idle timeouts and no per-request cost. Client-side: after auth, hand the client a specific gateway address or use a consistent-hash LB so that reconnects tend to land on the same server, which keeps subscription churn low. Cloud LBs need explicit idle-timeout configuration or they will drop sockets at 60 seconds; heartbeats every 30 seconds keep them alive.

Message ordering and IDs

How do all clients agree on the order of messages in a channel, without a single global sequencer?

The requirement is total order within a channel and nothing across channels. That is the key relaxation: it means we never need a global counter, only something that is monotonic per channel or, more loosely, time-ordered with unique tiebreaks.

The id does three jobs: unique identity for dedupe, sort key for display and storage, and cursor for resync ("give me everything after X"). A good id design makes all three trivial; a bad one (auto-increment per table, random UUID) makes at least one of them painful.

  • Time-ordered 64-bit id (Snowflake) chosen
  • Per-channel sequence number situational: when gap detection matters more than write throughput, e.g. financial audit trails
  • Database timestamp / auto-increment rejected

The answer: Snowflake-style ids: 41 bits of milliseconds, 10 bits of node id, 12 bits of per-node sequence. Uniqueness without coordination, time order good enough for humans, and a natural cursor. Within a channel, Kafka partitioning by channel_id then gives a single consumer order for fan-out, so all online clients receive the same sequence. Clients sort by id, which resolves the rare case where two messages are delivered out of order. If the interviewer pushes on strictness, describe the per-channel sequence as an upgrade for specific channels rather than the default.

Two users send at the same millisecond from different regions. Who wins and does it matter?

The Snowflake id breaks the tie by node id and sequence, so there is a deterministic order but it is arbitrary with respect to real time. For chat that is fine: both users perceive their own message as sent first and there is no causal dependency between them. If the interviewer pushes, causal order is what matters: a reply must follow the message it replies to, and that is guaranteed because the reply is created after the original was received and acked.

Clock skew: a Message service node's clock is 5 seconds ahead. What breaks?

Its ids sort 5 seconds into the future, so its messages appear after messages sent slightly later by other nodes. Worse, cursors: a client that has seen the skewed id will miss messages from correct nodes with smaller ids on resync. Mitigations: NTP with tight bounds, refuse to start if skew exceeds a threshold, never let the sequence go backwards on the node, and have resync fetch by id but also overlap by a few seconds of time to catch skew.

How do you paginate history with these ids? What about jumping to a specific date?

Cursor pagination: 'give me 50 messages in channel C with id less than X'. Because ids are time ordered and the clustering key is id descending, that is a single range scan. Jumping to a date is the same query: construct the minimal Snowflake id for that timestamp, which is just the timestamp bits with zeros, and query from there. No offset pagination anywhere.

If ordering per channel is the requirement, why route through Kafka at all? Cassandra already stored it.

Kafka gives every downstream consumer the same ordered stream without each of them polling the DB, and partitioning by channel id means the fan-out consumer processes a channel's events in order on one thread. It also decouples write latency from the number of consumers: adding search or analytics does not add latency to send. The order in Kafka matches id order in practice because a single channel's writes are serialised by the partition.

Message storage and partition key

Which database holds years of messages, and what is the partition key so history reads stay fast as channels age?

Access pattern: append-heavy writes, reads are almost always "most recent N in channel C" or "N around message id M in channel C". Cross-channel queries do not exist on the hot path; search is served by a separate index. This is the textbook shape for a wide-column store.

The subtle part is the partition key. Partitioning by channel_id alone creates unbounded partitions: #general after five years is tens of millions of rows in one partition, which Cassandra handles badly. Adding a time bucket (day or week depending on volume) bounds partition size while keeping a page of history inside one or two partitions.

  • Cassandra / Scylla, partition (channel_id, time_bucket) chosen
  • Sharded Postgres / MySQL by channel_id situational: the team already runs a sharded SQL platform at scale, or needs relational features on messages (threads with joins, complex permissions)
  • Single relational database rejected: a prototype

The answer: Cassandra-style wide-column store with partition key (channel_id, time_bucket) and clustering key message_id DESC. Bucket size is chosen per channel volume so a partition stays under ~100 MB. Reads for "latest page" hit the current bucket and, if short, the previous one. Writes use LOCAL_QUORUM for durability within the home region, with async cross-region replication. Metadata (users, channels, memberships) stays relational in Postgres because it needs constraints and joins and is orders of magnitude smaller. Mention that Slack chose sharded MySQL and explain why both are defensible: the interviewer wants to hear you match the store to the access pattern, not name a brand.

A partition for (#general, today) is getting hot: 10k writes/sec into one partition. What do you do?

Shrink the bucket for that channel to an hour, or add a small hash suffix to the partition key for hot channels so writes spread across N partitions, reading N partitions in parallel and merging by id. Choose bucket size per channel from observed volume and store it in channel metadata. Also confirm the real bottleneck: 10k writes/sec to one Cassandra partition is survivable briefly but hurts compaction.

How do you handle message edits and deletes in an append-only, LSM-based store?

Edits are an upsert on the same primary key, which LSM handles as a new version. Deletes write a tombstone. The problem is tombstone accumulation making reads slow in channels with heavy deletion; mitigations are soft deletes (a deleted flag) that are filtered at read time and purged by a periodic job, and tuning gc_grace_seconds. Edit history, if needed, is a separate table keyed by message id.

Why not put messages in Postgres like the metadata? One database is simpler.

Volume and access pattern. Messages are hundreds of TB, append-only and read by range on one key; metadata is gigabytes, relational and needs constraints and joins. A B-tree with billions of rows in one table needs manual sharding and its write amplification on a busy channel index is poor compared to LSM. Slack does run sharded MySQL and it works, but they invested heavily in Vitess; for a fresh design the wide-column store matches the pattern with less operational invention.

Multi-region: a workspace homed in the US has members in Europe. Where are writes and reads served?

Writes go to the home region for the channel to preserve a single order and LOCAL_QUORUM durability. Reads of recent history can be served from an async replica in the reader's region with a few hundred ms lag, which is acceptable for scrollback. Real-time delivery crosses regions via Kafka mirroring or the gateway in Europe subscribing to the US pub/sub tier. The cross-region latency shows up as roughly 100 ms extra on delivery, still under the 1 s cross-region target.

What is the retention story? Some workspaces want 90 days, some want forever.

Cassandra TTL per row is the simple mechanism but cannot be changed retroactively for existing rows. Better: store retention policy on the workspace and run a deletion job per time bucket, dropping whole partitions, which is cheap. Legal hold overrides deletion. Search index deletions must follow the same policy or the index becomes the leak.

Delivery guarantees, acks and dedupe

What does "never lose a message" actually mean, and where can duplicates come from?

Exactly-once delivery over a network is not achievable in general; what you can build is at-least-once delivery plus idempotent processing, which is indistinguishable to the user. Every hop in the send flow can time out after doing its work, and every retry at that hop produces a potential duplicate. The design places a dedupe key at each place a retry can occur.

Three retry points: client → server (socket drop before ack), server → Kafka (publish timeout after DB write), and server → client (gateway crash mid-push). Each is handled differently.

  • At-least-once + idempotency at each hop chosen
  • At-most-once (fire and forget) rejected: ephemeral signals only: typing indicators, presence
  • Transactional exactly-once end to end rejected

The answer: At-least-once everywhere, idempotent everywhere. Client sends carry a client_msg_id; the Message service keeps (user_id, client_msg_id) → message_id in Redis for a few minutes and returns the existing id on a retry. The DB write is the durability point; the Kafka publish is made reliable with a transactional outbox (or CDC from the messages table) so a crash between DB write and publish still emits the event. Downstream, every consumer keys on message_id, and clients dedupe on it. Reconnect resyncs from per-channel cursors, which covers any delivery lost while a gateway was dying. Typing and presence are explicitly at-most-once because losing them costs nothing.

The client sends, the server writes to Cassandra, then the server crashes before acking. What does the user see and what happens next?

The client's send times out and it retries with the same client message id. The dedupe entry in Redis (or the DB) returns the existing message id, so no duplicate is written, and the ack arrives. Meanwhile the outbox or CDC has already emitted the event, so recipients got the message once. The user sees a spinner for a few seconds and then a checkmark.

Your dedupe store is Redis with a TTL. What if the retry comes after the TTL?

Then a duplicate can be written. Choose the TTL longer than any plausible client retry window, say 10 minutes versus a client that gives up after 1. For belt and braces, make the dedupe key part of the Cassandra row via a lightweight transaction or a secondary table keyed by (user, client id) with a longer TTL. At Slack's scale the Redis approach with a generous TTL is what is actually used.

How does the receiving client know it missed a message rather than there simply being none?

It cannot from a single message, since ids are not gapless. It knows on reconnect by comparing cursors, and it can periodically ask 'latest id in channel C' for open channels and compare to what it has. Read receipts and unread counts come from a server-side counter, so the badge is correct even when the realtime stream dropped something.

Is there any part of this system where at-most-once is the right choice?

Typing indicators, presence heartbeats and ephemeral cursors. They are superseded within seconds and losing one has no user-visible consequence. Making them durable would add load to Kafka and the DB for zero benefit. Being explicit about which signals are lossy is a mark of a mature design.

Presence at scale

Why is presence the feature most likely to take down a chat system, and how do you make it safe?

Presence is deceptively expensive. Ingest is fine: 1 M heartbeats per 30 s is 33 k tiny writes per second. Broadcast is the trap: if every status change is sent to everyone who might see it, a 50 k-user workspace produces 50 k events per change × 50 k viewers. Naive presence is quadratic.

The way out is to admit presence is a hint. Nobody needs to know within 100 ms that a colleague went idle. That permits TTL-based expiry instead of explicit offline events, coalescing, and subscribing only to what is on screen.

  • TTL keys + subscribe-to-visible + coalescing chosen
  • Broadcast every change to the whole workspace rejected: workspaces under ~100 users
  • Poll presence on demand situational: presence for rarely viewed surfaces, e.g. a member directory

The answer: Heartbeats refresh a Redis key with a 60 s TTL; expiry means offline, so no cleanup path is needed for crashes. Clients subscribe to presence only for user ids currently visible (DM sidebar, current channel's member list), and the Presence service publishes deltas to those subscriptions through the same pub/sub used for messages. Changes per user are rate limited (one per few seconds) and coalesced, and last-write-wins resolves races between devices. Presence is explicitly at-most-once and served stale under load: when the cluster is stressed, presence degrades first, by design, and messaging keeps working.

A user has 5 devices. What is their presence?

Presence is per user, computed as the max over devices: online if any device is active, away if all are idle, offline if all keys expired. Each device refreshes its own key (user, device) and a small aggregation on read or on change computes the user-level status. Last-write-wins per device avoids races between them.

The Redis cluster holding presence dies. What breaks?

Presence and typing degrade: everyone looks offline or stale until the cluster is back and heartbeats repopulate it, which takes one heartbeat interval. Messaging continues because the connection registry for fan-out should be a separate keyspace with replication, or at least the pub/sub path should survive. This is the argument for keeping presence in its own cluster: it is the component you are most willing to lose.

How does the client decide which users to subscribe to for presence?

Whatever is on screen: the DM sidebar, the member list of the open channel, and the authors of visible messages. Subscriptions are diffed as the UI changes, with a small delay to avoid churn while scrolling. Cap the subscription set at a few hundred; beyond that, show presence only on hover via a point query.

What does a 'presence storm' look like and how do you prevent it?

An office loses Wi-Fi for 30 seconds: 5,000 users go offline then online within seconds, each change fanning to hundreds of viewers, millions of events. Prevention: debounce status changes, do not broadcast offline until the TTL actually expires rather than on socket close, coalesce multiple changes per user per few seconds, and rate limit presence deliveries per receiving connection. Users will not notice a 10 second delay in a green dot.

Related