Routing messages across WebSocket gateways with broadcast, a connection registry or pub/sub: an interactive model

A chat service keeps each user’s WebSocket open on one of many gateway servers. When someone sends a message, the server handling it has to get it to whichever gateways hold the recipients’ connections. Broadcasting to every gateway is simplest and stops scaling almost at once. A connection registry or pub/sub channels send each message only where it is needed, at the price of keeping routing state up to date as users connect, disconnect and move between gateways.

The playground connects a million users across a fleet of gateways and sends 20,000 messages a second, either one-to-one or to groups of 50.

What the numbers say

• Broadcast does not scale out. Every gateway reads every message, 20,000 a second, whether there are 10 gateways or 500. With 50 gateways only 2% of what each one receives is for its own users, and the cluster moves a million messages a second to deliver 20,000.
• Targeted routing scales with the fleet. With a registry or per-user channels each message goes to one gateway, so each of 50 gateways handles 400 a second, and adding gateways keeps reducing it. The registry costs a lookup per message; pub/sub keeps the same mapping as broker subscriptions.
• Groups change the picture. A group of 50 spread over 50 gateways touches about 32 of them, so broadcast is less wasteful, 64% useful, and looking up every member costs a million registry lookups a second. Per-room channels, with each gateway subscribing once for all of its members in that room, deliver to just those gateways without per-member lookups.
• Routing state must heal. Registry entries carry a time to live refreshed by heartbeats, so a crashed gateway’s users expire and reappear when they reconnect elsewhere; messages for users who are not connected anywhere are stored and trigger a push notification.

The moving parts

# on connect: the gateway records where the user is (registry) or subscribes (pub/sub)
SET conn:user:812 gw-17 EX 90          # refreshed by heartbeats, expires if the gateway dies
SUBSCRIBE user:812                     # or: SUBSCRIBE room:team-eng, once per room per gateway

# sending a direct message
gw = GET conn:user:812                 # registry: one lookup...
POST http://gw-17/deliver {to: 812, ...}   # ...then one targeted send
PUBLISH user:812 {...}                 # pub/sub: the broker knows who subscribed

# if the user is not connected anywhere: store it and send a push notification

In a system design interview

For chat, presence or live updates, say how a message finds the right connection: “Gateways hold the WebSockets and register each user in a connection registry with a TTL, refreshed by heartbeats. The chat service looks up the recipient’s gateway and forwards the message directly; for channels with many members, gateways subscribe to the channel on a pub/sub layer once for all their local members. Offline recipients get the message from storage on reconnect and a push notification now.” Expect the follow-ups: users connected on several devices, a gateway crash in the middle of a delivery, and ordering of messages within a conversation.

Read the guide to WebSockets vs SSE vs long polling or group chat fan-out, see real-time delivery and online presence, or watch a strong candidate design WhatsApp.

Questions people ask