Real-time systems: WebSockets, SSE and push

Long polling vs Server-Sent Events vs WebSockets, connection servers and registries, pub/sub fan-out, presence, reconnection and resync, and scaling to millions of connections.

Chat, notifications, live dashboards, collaborative editing, multiplayer games and ride tracking all need the server to push data to the client the moment it changes. HTTP was built the other way round. The design questions are how the push channel works, how a message finds the right connection among millions, and what happens when the connection drops.

Transport options

Short polling. The client asks every N seconds. Trivial, wasteful, latency up to N. Acceptable for a status page that changes rarely.

Long polling. The client asks and the server holds the request open until there is data or a timeout (say 30 s), then the client immediately asks again. Works through every proxy and firewall, near-real-time, but each message costs a full HTTP round trip and servers hold many open requests.

Server-Sent Events (SSE). One long-lived HTTP response that streams text/event-stream messages from server to client. Simple, auto-reconnects with Last-Event-ID, works over HTTP/2 multiplexing, but one-directional: the client sends via ordinary requests. Ideal for notifications, feeds, and streaming AI responses.

WebSockets. A persistent bidirectional TCP connection upgraded from HTTP. Lowest latency and overhead per message, the right choice for chat, games, collaborative editing, and anything with frequent client-to-server messages. Costs: stateful servers, load balancers must support the upgrade and long connections, and you own the reconnection logic.

Default: WebSockets for bidirectional, SSE for server-to-client only, long polling as a fallback. Mobile background delivery is a separate channel through APNs and FCM because the OS kills sockets.

The connection tier

Long-lived connections need their own tier of connection servers (gateways) whose only job is holding sockets, authenticating them, and forwarding frames. They are stateful in exactly one way: they know which users are connected to them. A server holds 100 k to 1 M idle connections depending on memory and heartbeat load; 10 M concurrent users is 20 to 100 servers.

Behind them, business services stay stateless. A message sent by user A goes to a message service over an ordinary RPC, is persisted, and then must be delivered to every connected member of the channel, wherever their sockets are. That is the fan-out problem.

Finding the connection: the registry

A connection registry maps user_id → connection server(s). Redis is the usual store: on connect, SADD conns:{user_id} server-17 with a TTL refreshed by heartbeats; on disconnect or server death, remove. To deliver to a user, look up their servers and forward. For large groups, a lookup per member is expensive; the alternatives:

• Pub/sub per channel. Each connection server subscribes (Redis pub/sub, Kafka, or a purpose-built broker) to the topics its connected users belong to. A message is published once per channel; every server with a subscriber receives it and forwards locally. No per-member lookup, and the subscription count is bounded by (servers × channels with a member on that server).
• Per-user delivery queues for reliable inbox semantics: each user has a queue, workers push into it, the connection server drains it. Heavier but gives at-least-once delivery.

Slack, Discord and Facebook Messenger all use a variant of the pub/sub approach with a registry for direct lookups.

Fan-out at scale

A channel with 50 k online members turns one send into 50 k deliveries. At 10 messages per second in that channel, that is 500 k frames per second from one source. Bound it: publish once per channel and let servers fan out locally (they already know their local subscribers); batch frames per server; and for very large channels, degrade features (no typing indicators, coalesced presence) so the per-message cost stays small. State the number: "fan-out is the largest write stream in the system, larger than the messages themselves".

Presence and typing

Presence (online/away) is the highest-volume, lowest-value signal. Every connect, disconnect and idle change is an event fanned out to everyone who can see that user; with large teams it exceeds message traffic by orders of magnitude. Design it lossy: heartbeats every 30 s to Redis with a 60-s TTL; presence changes batched per few seconds and only delivered for users currently visible on screen (the client subscribes to the presence of the 50 users in view, not all 10 000 in the workspace). Typing indicators are the same, with a 3-second TTL and no persistence.

Ordering, acknowledgement and resync

Sockets drop: mobile networks, laptop lids, deploys. Design for it as the normal case.

• Ids are the cursor. Every message has a time-ordered id per channel. The client tracks the last id it has seen per channel.
• Acknowledge. The server acks a client send with the assigned id; the client marks its optimistic message as sent. Unacked sends are retried with the same client_msg_id so the server deduplicates.
• Resync on reconnect. The client sends its last-seen ids; the server returns everything after them from the store (the gap), then resumes streaming. If the gap is too large (offline a week), the client refetches recent history instead.
• At-least-once plus dedupe is the delivery guarantee. Exactly-once over a socket is not achievable; idempotent clients make it unnecessary.

Heartbeats and half-open connections

TCP does not notice a dead peer for a long time. Both sides send pings (every 20 to 30 s) and close the connection after two missed pongs. The server uses the same interval to refresh the registry TTL, so a crashed server's entries expire on their own. Keep the ping interval short enough for NAT and load balancer idle timeouts (often 60 s).

Load balancing and deploys

WebSockets need an L4 or upgrade-aware L7 balancer with long idle timeouts, and a routing decision that is sticky by nature (the connection stays where it landed). Deploying connection servers means draining: stop accepting new connections, tell clients to reconnect (a GOAWAY-style frame) over a few minutes so they do not all reconnect at once, then stop. A restart without draining reconnects all users in one second, which is a thundering herd on auth and the registry; stagger with client-side jitter.

In the interview

"Clients hold a WebSocket to a connection-server tier, 500 k connections per server. A registry in Redis maps user to server with a heartbeat-refreshed TTL. Sends go over RPC to the message service, which persists, then publishes once per channel to a pub/sub layer that connection servers subscribe to for their local users, so fan-out cost is per-server, not per-member. Each message has a time-ordered id; clients resync from their last seen id on reconnect, and retries carry a client message id for dedupe. Presence is lossy: 30-second heartbeats, delivered only for users on screen."