Design Nearby Friends

Show each user which friends are within 5 miles, refreshed every 30 s, for 10 M concurrent users, without leaking anyone’s location.

Last updated 2026-10-27. Difficulty: hard. Patterns: real-time, pub-sub, fan-out, geospatial, privacy. Reported at Meta, Snap, Google, Uber, Lyft, Grab.

Functional requirements

• Opt in to share your live location with friends. Off by default. While the app is open it reports a position about every 30 s; with a second opt-in, the OS sends occasional background updates. Sharing is between friends only, never with strangers.
• See which friends are nearby. Friends within a radius the viewer chooses (5 miles by default), sorted by distance, each with the distance and how fresh it is ("updated 20 s ago").
• Live refresh while the app is open. Entries move as friends move, about every 30 s. A friend who stops reporting drops off within 10 minutes without anyone deleting anything.
• Pause sharing. One switch, effective everywhere within a couple of seconds: friends see you disappear at once, not when a timer runs out.
• Choose who sees you, and block. All friends, or a chosen list. A block hides both people from each other, in the live list and in everything the other could read.
• Reduced precision. Per friend or for everyone: they see an approximate area (a fixed ~5 km cell) and a rounded distance, never your point. The server guarantees it, not the viewer’s app.
• Optional location history. Opt-in, kept for later features such as a timeline. The user can view it, choose a retention period and delete all or part of it, and a deletion must reach every copy, backups included.
• Out of scope. Map rendering, "Alice is nearby" notifications, ETAs and routing, the friend graph itself (an existing service we read), and showing strangers nearby, which we use as the contrast where a geo index is the right tool.

Non-functional requirements

• Scale (10 M concurrent · ~400 friends each). 1 B users, 100 M use the feature daily, 10 M have it open at the evening peak. Friend lists average ~400 and are capped at 5,000.
• Freshness (interval 30 s · server path p99 < 1 s). A friend’s move appears within one reporting interval plus under a second through our servers. This is the requirement that drives the hardest trade-off: halving the interval doubles both the battery cost on every phone and the fan-out, which is already the largest stream in the system.
• Battery and data (≤ 1 radio wake per interval). People uninstall location apps that drain the battery. Uplink and downlink share one radio burst, the interval stretches when the phone is still, and the background uses the OS’s low-power location service.
• Privacy enforced on the server (block and pause effective < 2 s). Audience, blocks, pause and precision are applied by our servers at publish and at subscribe, never by trusting the viewer’s app. A coarse viewer never receives a precise coordinate, not even inside a payload the app would hide.
• Availability (99.9 % · degrade to stale). Losing a location update is harmless, because the next one arrives within 30 s; losing a privacy change is not. So location delivery is best effort, and privacy changes are acknowledged and retried.
• Ephemeral live state, durable history (latest: TTL 10 min · history: opt-in, RF 3). The latest location is a cache that can be lost and rebuilt within one interval. History is the only durable record of where someone has been, so it exists only for users who turned it on.
• Deletion (reads < 1 min · backups < 30 days). Deleted history disappears from every read path within a minute and from backups within the stated window. Per-user encryption keys make the backup copies unreadable as soon as the key is destroyed.

Back-of-envelope estimates

• Location updates in: ~333 k/s peak · ~115 k/s avg. 10 M concurrent ÷ 30 s = 333 k/s at peak, an upper bound because still phones report less often. On average ~3.5 M users have it open (100 M daily × 5 sessions × 10 min ÷ 1 440 min): ~115 k/s. At ~100 B each the ingest is only ~33 MB/s at peak. Writes are not the hard part.
• Deliveries out of pub/sub: ~13 M/s · ~1.3 M/s reach a phone. Assume ~10 % of a user’s 400 friends are active at the same time at peak (friends share time zones and habits): 40. Each update is delivered to the servers of those 40: 333 k × 40 = 13.3 M deliveries/s. About 1 in 10 of those friends is within 5 miles, so ~1.3 M/s become pushes to a phone. Fan-out is the largest stream in the system, 40 times the updates.
• Subscriptions held: ~4 B · ~400 GB · 90 % silent. Every active user’s server subscribes to the channels of all ~400 friends, online or not: 10 M × 400 = 4 B subscriptions, at ~100 B of broker bookkeeping each ~400 GB. Only the 10 % to active friends carry traffic; the other 3.6 B are channels nobody publishes on, which cost memory and nothing else.
• Pub/sub nodes: ~100 primaries (CPU-bound). 13.3 M deliveries/s ÷ ~130 k/s per Redis-class node, kept under ~50 % CPU = ~100 nodes, each with a replica. Memory per node is 400 GB ÷ 100 = 4 GB, nowhere near a limit: deliveries size this tier, not subscriptions.
• Connection servers: ~400 at 25 k sockets each. 10 M connections ÷ 25 k per server = 400 servers, budgeted low because each socket brings work: 25 k × 40 ÷ 30 s = 33 k deliveries/s in per server, as many distance checks (under a microsecond each), ~3.3 k pushes/s out and ~830 publishes/s. The local subscription map is 25 k × 400 = 10 M entries, ~600 MB.
• Latest-location cache: ~1 GB · 333 k writes/s · ~7 M key reads/s. One entry per user who reported in the last 10 minutes, ~10 M at peak × ~100 B = ~1 GB. Writes are 333 k/s. Reads come from app opens: ~17 k/s at peak (next row) × 400 friends = ~6.7 M keys/s, sent as one MGET per shard. The cache is sized by operations, not memory: about 20 shards with replicas.
• Connect churn at peak: ~17 k opens/s · ~7 M subscribes/s. Little’s law: 10 M concurrent ÷ ~600 s per session = ~17 k connects/s. Each subscribes to ~400 channels: ~6.7 M subscribes/s, and as many unsubscribes as sessions end. That is half the delivery load again, which is why a session survives a quick app switch instead of being torn down.
• History volume: ~10 B points/day · ~500 GB/day. 100 M daily users × 5 sessions × 10 min × 2 updates/min = 10 B points/day, the same as 115 k/s × 86 400. At ~50 B per stored point that is ~500 GB/day, ~180 TB a year before replication, if every user turned history on. The opt-in, downsampling of still periods and retention limits make the real number smaller.

Components

• Mobile app (WebSocket · OS location APIs): Reads the position from the OS and reports it about every 30 s while open, stretching the interval when the phone is still and using the OS’s significant-change service in the background. Holds one WebSocket and renders the nearby list from pushes, computing "updated 20 s ago" from server timestamps. It enforces nothing: it only ever receives what this viewer is allowed to see.
• Edge load balancer (TLS · least connections): Terminates TLS and places new WebSocket connections by least connections, because sessions last minutes and round robin would leave newly added servers empty. Honours a resume hint so a quick reconnect lands back on the server that still holds the session. Routes the REST settings and history calls to the sharing service.
• Connection servers (stateful · 25 k sockets each): ~400 servers, each holding one WebSocket per active user plus that session’s state: the viewer’s latest position, their audience, and the last position of each active friend. On an update they apply the sharer’s settings, write the cache, publish once and append history; on each delivery from a friend’s channel they check permission and distance for the local viewer and push only what changed. Broker subscriptions are reference-counted per server, so two local viewers with a common friend cost one.
• Session registry (Redis · user → server, epoch): Maps a user to (server, connection, epoch), written on connect with an epoch from INCR, so a reconnect always supersedes the session before it. Instead of 10 M per-user TTLs refreshed by heartbeats, each server refreshes one liveness key every 5 s, and an entry that points at a dead server reads as absent. Read only for revokes and duplicate-session kicks, never on the location path.
• Location cache (Redis · TTL 10 min): The latest location per sharing user: loc:{user} holds the point, the coarse cell and the server timestamp, rewritten with EX 600 on every update, so users who stop reporting fall out without a cleanup job. ~1 GB in ~20 shards with replicas, sized by the ~7 M key reads a second that app opens cause at peak. Losing it costs one reporting interval, because every active user rewrites their entry within 30 s.
• Pub/sub tier (Redis pub/sub · ~100 primaries): One channel per sharing user, loc:{user}, plus loc:{user}:c carrying only the coarse cell. ~100 independent Redis primaries with replicas, placed on a consistent-hash ring by channel, so an update is one PUBLISH on one node delivered to the ~40 servers subscribed. Fire and forget on purpose: a lost delivery is replaced within 30 s, and anything that must not be lost travels another way.
• Tier config (etcd · ring vN · interval): The versioned pub/sub ring (primaries and their virtual nodes) and the fleet-wide reporting interval, watched by every connection server. A resize is a new ring version moved through stages (dual subscription, publisher switch, retirement), never an edit in place. Small, rarely changed and needing ordered, confirmed transitions across 400 servers, which is what a consistent store with watches is for.
• Load governor (control loop · every 10 s): Watches pub/sub CPU, delivery lag and connection-server queues every 10 s and raises the reporting interval in tier config when they cross a threshold, 30 s to 60 s for moving phones. Steps back down slowly with hysteresis so the fleet does not oscillate. The interval is the one lever that cuts updates, deliveries and battery use together.
• Sharing service (audience · settings · revokes): Decides who may see whom. A viewer’s audience is the friends whose sharing is on and not paused, whose audience includes the viewer, with no block either way, plus the precision each allows, cached per viewer with a version. On any change it writes the store first, then sends acknowledged revokes to the affected sessions found in the registry. Also serves the settings and history-deletion APIs.
• Sharing store (MySQL · sharded by user): The source of truth for privacy: sharing on or paused, default precision, audience mode, per-friend grants and blocks, each change versioned. A few hundred bytes per user and strongly consistent, because a lost or reordered block is an incident. Read mostly through the sharing service’s cache.
• Friend graph (existing service · cached): The company’s existing social graph: who is friends with whom, capped at 5,000 friends a person. We read it to build each viewer’s audience and invalidate cached audiences on its friendship-change events. It is the filter of this whole design: we never search by geography, only by this list.
• Update stream (Kafka · key = user · 3 days): Updates from users with history on, appended by the connection server after the publish, keyed by user id so one user’s points stay ordered on one partition. ~20 MB/s at peak. Three days of retention lets the history writer catch up after an outage and also bounds how long a deleted point can linger here.
• History writer (batched per user-day): Consumes the update stream and writes points in batches per user and day, minutes behind real time, which nobody notices. Drops points within ~20 m of the previous one while a user is still, encrypts each point under that user’s history key, and skips anything older than a user’s deletion watermark. Idempotent on (user, server timestamp), so a replay after a crash writes nothing twice.
• Location history (Cassandra · (user, day)): Append-only points partitioned by (user_id, day) and clustered by time: a day of one user is one partition, a range delete is a few partition deletes, and retention is a TTL. ~500 GB a day at the upper bound before replication factor 3. Points are encrypted under per-user keys held in a key service, so destroying a key makes every copy, backups included, unreadable at once.

User flows

1. Open the app: first screen, then the stream. The start of a session. Alice must see her nearby friends within a second, although they will not report again for up to 30 s, and an update that lands while her list is being built must not be lost.
   1. The app opens a WebSocket through the load balancer and sends a hello with its first location fix. Least-connections placement, because sessions last minutes and a freshly added server would otherwise stay empty. The hello carries the first fix so distances can be computed at once instead of after the next 30 s cycle. If it carries a resume token for a server that still holds the session, the load balancer routes it there and the session reattaches without steps 2 to 5.
   2. The connection server registers the session under a new epoch. INCR gives an epoch that only grows. If Alice still has an older session on another server, that server is sent a kick and closes its copy instead of both publishing her location. Entries carry no per-user TTL: the owning server’s liveness key, refreshed every 5 s, decides whether an entry is live, which saves ~330 k TTL refreshes a second.
   3. The server asks the sharing service for Alice’s audience: the friends who share with her, and at what precision. The subscribe-side privacy check. A friend who paused, who shares with a list Alice is not on, or who is blocked in either direction is simply absent from the answer, so her server never subscribes to them or reads their location. The answer is cached per viewer with a version and rebuilt only when a friendship or a friend’s settings change.
   4. The server subscribes to the channel of every friend in the audience, before reading anything. Subscribe first, read second: an update that lands while the snapshot is being read is then in the snapshot or on the channel, never lost in between, and the newer timestamp wins. Channels are grouped by owning node from the ring in tier config, so ~400 channels over ~100 nodes is ~100 small SUBSCRIBE commands sent in parallel. Offline friends are subscribed too; their channels are silent and cost only memory.
   5. The server batch-reads those friends’ latest locations from the cache. One MGET per cache shard, ~20 in parallel, a few milliseconds. About 40 of the 400 keys exist, the friends who reported in the last 10 minutes; the rest have expired, which is how the TTL answers "who is active" without anyone tracking it. For a coarse friend the server keeps only the cell from the entry and discards the point.
   6. The server computes distances, keeps the friends inside Alice’s radius and sends the first screen. Forty distance checks take microseconds, so the list appears about 100 ms after the hello rather than after the next cycle. The server keeps those 40 positions in memory, so when Alice herself moves, her list is recomputed locally with no further reads.
   7. From now on, deliveries on the subscribed channels keep the list live. A friend who comes online later needs no new subscription: Alice’s server is already listening on their silent channel, so their first update simply arrives. Alice’s own updates write the cache and publish on her channel like everyone else’s.
2. Bob moves: one publish, forty deliveries, four pushes. The core path, 333 k times a second at peak. An update is written once and published once, and the work that depends on the viewer (permission, distance, radius) runs on each viewer’s own server.
   1. Bob’s app sends a location update over its WebSocket. About 100 bytes every 30 s while he moves. The server timestamps it on receipt and rejects impossible jumps (faster than an airliner since the last fix), which stops cheap spoofing. The update doubles as the heartbeat, so a moving phone sends nothing else.
   2. Bob’s server applies his settings: paused stops here, and if anyone sees him coarsely it also computes his cell. Publish-side enforcement. Coarse means the fixed ~5 km geohash cell containing Bob, not his point plus random noise, because noise averages away over a hundred updates and a fixed cell says the same thing every time. His settings live in his session, loaded at connect and changed by revokes, so there is no lookup per update.
   3. The server overwrites Bob’s entry in the location cache with a 10-minute TTL. An overwrite, not an append: only the latest point matters here. The TTL is twice the longest interval a still phone uses (5 minutes), so someone who is merely sitting still never falls out, and someone who closed the app is gone within 10 minutes without a cleanup job.
   4. The server publishes the update once on Bob’s channel, and once more on his coarse channel if he has coarse viewers. The ring maps loc:{bob} to one of ~100 nodes, which delivers to every connection server subscribed: the servers of Bob’s ~40 active friends. Bob’s server neither knows nor cares who they are. A second channel for coarse viewers is the price of never sending his point to a server that only needs his cell.
   5. If Bob has history on, the server appends the point to the update stream. After the publish, never in front of it: the live list must not wait for history. Users with history off never reach this step, so no record of their movements exists beyond a cache entry that expires in 10 minutes.
   6. Each subscribed server checks, for its local viewers of Bob, that they may still see him, and computes the distance. The viewer’s latest position is already in memory, so this is one distance calculation, under a microsecond; the publisher would need 40 cache reads to do the same. Each viewer has their own radius and precision, which only their own server knows. A delivery not newer than the last one seen from Bob is dropped, so a late message cannot move him backwards.
   7. Only viewers for whom Bob is inside the radius, or has just left it, get a push. About 4 of the 40 deliveries become pushes. Hysteresis stops a friend on the boundary from flickering: enter at 5 miles, leave beyond 5.5. A viewer’s pushes are held for a few seconds and sent together, preferably right after that viewer’s own update, so their radio wakes once per interval.
   8. Minutes later, the history writer batches Bob’s points into his partition for the day. Batches per partition make the write cheap, and nobody reads history in real time, so minutes of lag cost nothing. Points within ~20 m of the previous one while Bob is still are dropped, which removes most of a night or a day at a desk.
3. New Year’s Eve: concurrency doubles. Around midnight twice the usual peak opens the app at once. Deliveries are updates times active friends, and both double, so fan-out grows four times. The defence is the reporting interval, which the server controls.
   1. Concurrency doubles to ~20 M within the hour, and opens spike as everyone checks who is nearby. Updates double, and each user now has ~80 active friends instead of 40, so deliveries reach ~53 M/s at the upper bound against a pub/sub tier that does ~26 M/s at full CPU. Fan-out grows with the square of concurrency, which makes a busy night more dangerous than a year of growth. The connection tier was scaled out the week before, because the date is on the calendar.
   2. The load governor sees pub/sub CPU and delivery lag cross their thresholds. Lag is measured, not guessed: every delivery carries its publish time, and connection servers report its age on arrival. The thresholds sit below the point where broker output buffers grow, because a broker that falls behind disconnects slow subscribers, and that turns overload into a reconnect storm.
   3. The governor raises the reporting interval in tier config. Moving phones go from 30 s to 60 s; still phones are already on 5 minutes. Updates and deliveries both scale with 1 / interval, so this one change halves the load within a cycle, to ~27 M/s at the upper bound and well under it in practice. Users barely notice: a friend’s position is up to a minute old instead of 30 s.
   4. Connection servers pick up the change and hand each phone the new interval in its next ack. No push to 20 M phones is needed: every phone is about to send an update and reads next_in_s from the ack, so the whole fleet is on the new interval within 30 s. Old app versions that ignore the field are throttled on the server: an update that arrives early is acknowledged and dropped.
   5. Servers coalesce pushes harder and drop the subscriptions of viewers who sent the app to the background. A backgrounded app shows nothing, so after the 2-minute grace its server releases that viewer’s reference counts and stops checking distances for them. Foreground viewers get at most one frame every 10 s instead of every few seconds. Both cut work without making any visible list worse.
   6. As load falls, the governor steps the interval back down, one step at a time. Hysteresis: relax only after 15 minutes below a lower threshold, and by one step, so the fleet does not flip between 30 s and 60 s every minute. Applied per region, the same lever protects one region’s midnight without slowing the rest of the world.
4. A connection server dies with 25 k users. The stateful tier’s worst ordinary day. 25 k phones lose their socket at once, the server’s subscriptions must disappear, and every user must get their list back without a thundering herd on the registry, the sharing service, the cache and the broker.
   1. ws-217 crashes; its broker connections close and its liveness key expires. Redis subscriptions belong to the connection that made them, so when the server’s ~100 broker connections close, its ~10 M subscriptions vanish by themselves: no cleanup job and nothing leaked. Its 25 k registry entries point at a server whose liveness key is gone within 15 s, so they read as absent.
   2. Phones notice the dead socket and reconnect with exponential backoff and jitter. A missing ack within ~10 s triggers the reconnect. Jitter spreads 25 k reconnects over ~30 s instead of one second; for a planned drain, the server sends a close frame with a suggested delay so the spread is chosen rather than random. The load balancer places them on the least-loaded servers.
   3. The new server writes the session with the next epoch. If ws-217 was only partitioned and comes back, it finds epoch 43 in the registry, newer than its own 42, and closes its copy instead of publishing Alice’s location twice. Epochs turn "which session is real" into a comparison of two numbers.
   4. It rebuilds the session: audience, subscriptions, then a snapshot from the cache. The same path as an app open, ~400 subscribes and ~400 key reads per user, so 25 k users cost ~10 M of each spread over 30 s, ~330 k/s, a few percent on top of the normal peak. Nothing is replayed: locations are state, not a log, so the latest entry replaces everything Alice missed.
   5. Alice’s friends notice nothing; her next update publishes from the new server. Publishers do not know where subscribers are and subscribers do not know where publishers are: both only know the channel. While Alice was away her cache entry stayed valid, so friends kept seeing her last position with an honest "updated 40 s ago".
   6. A revoke that was in flight to ws-217 is re-sent to wherever Alice is now. A block Bob issued during the outage was sent to ws-217 and never acknowledged, so the sharing service re-reads the registry and retries against ws-090. Losing it would also have been harmless here: the audience ws-090 just fetched already excludes Bob, because the store was written before any revoke was sent.
5. Block, pause, and delete my history. The guarantees users actually care about. Each change is written to the store first and then pushed, with acknowledgement, to the servers that enforce it, because unlike a location update, a lost privacy change is an incident.
   1. Bob blocks Alice; the sharing service writes the block and bumps both audience versions. The store is written first, so any session built after this moment excludes Bob, whatever happens to the messages that follow. A block is symmetric: Alice disappears from Bob’s list as well as Bob from hers.
   2. It finds both live sessions in the registry and sends each server a revoke. Alice’s server releases its references to Bob’s channels, forgets his position and pushes "left"; Bob’s server does the mirror image. Revokes are RPCs with acknowledgement and retries, not pub/sub messages, because pub/sub may lose things and this must not be lost. The target is under 2 s end to end.
   3. Until the unsubscribe lands, Alice’s server drops anything from Bob by checking her audience version. Every delivery is checked against the viewer’s audience before the distance check, so a message already in flight on the old subscription is discarded. Enforcement never depends on the app: Alice’s phone receives nothing about Bob, rather than receiving it and hiding it.
   4. Carol pauses sharing; the sharing service records it and tells her server. Same order as a block: store first, so new sessions anywhere exclude her, then an acknowledged message to her live session. Pausing is per user rather than per pair, so it changes what her own server publishes instead of what each viewer’s server accepts.
   5. Her server stops publishing, deletes her cache entry and publishes a tombstone. Without the tombstone her friends would keep seeing her last position until the 10-minute TTL ran out, which is not what pause means. The deleted entry is the guarantee, because a friend who opens the app now reads nothing; the tombstone is best effort and simply makes the live lists drop her at once.
   6. Dave deletes his whole history; the service deletes his partitions and destroys his history key. A deletion watermark makes the history writer skip his points still in the 3-day stream. Partition deletes for each (user, day) remove the data from every read within seconds and compaction removes the bytes within days. Destroying his per-user key makes backup copies unreadable at once, and the backups themselves expire within 30 days, which is exactly what he is told.

Deep dives

The theory behind it

• Location tracking at scale: Ingesting and serving live GPS locations for drivers, couriers and devices: update frequency and battery, the ingest path, latest-location stores versus history, geospatial indexing of moving objects, sharing a live location with riders, map matching, geofencing, privacy and retention.
• Real-time systems: WebSockets, SSE and push: Long polling vs Server-Sent Events vs WebSockets, connection servers and registries, pub/sub fan-out, presence, reconnection and resync, and scaling to millions of connections.
• Presence and managing millions of connections: Holding millions of persistent WebSocket connections and showing who is online: connection gateways, routing messages to the right server, heartbeats, presence with TTLs, fan-out of status changes, typing indicators, reconnect storms and deploys.

Related

• Design Slack
• Design a News Feed
• Design Google Maps
• Design Uber
• All 83 system design interview questions
• Run this question for a friend: the interviewer kit