Design Google Docs

• Multiple people edit one document at the same time — Edits appear for everyone within a couple of hundred milliseconds, and every client ends up with byte-identical text regardless of the order updates arrived.
• Local edits are never blocked — Typing applies to the local copy immediately and is sent in the background. A design that waits for the server before showing a character is unusable, which is what forces the conflict-resolution machinery.
• Presence and cursors — Who is in the document, where their cursor is and what they have selected. Ephemeral, lossy and high frequency: never stored, never allowed to block an edit.
• Offline editing and reconnect — Edits made while disconnected are queued locally and merged on reconnect without a manual conflict dialog.
• Version history and named revisions — Browse and restore earlier versions, see who changed what. This is the durable artefact; the live editing session is not.
• Comments and suggestions anchored to text — An anchor must survive concurrent edits around it, which is a different problem from anchoring to a character offset.
• Sharing and permissions — Owner, editor, commenter, viewer, plus link sharing. Permission changes take effect on the live session, not just on the next open.
• Rich text, not plain text — Bold, headings, lists, tables and images. Formatting is a concurrent operation too: two people bolding overlapping ranges must converge.
• Out of scope — Real-time video and chat in the document, the full editor UI, spreadsheet formula evaluation, and offline-first desktop sync clients.

• Edit propagation latency (p95 < 200 ms same region) — Beyond about 300 ms, collaborators start colliding because they no longer see each other in time. This is the requirement that keeps sessions pinned to one region.
• Local echo latency (< 16 ms) — A keystroke renders on the next frame, always. The server is never on the path of showing your own character.
• Convergence (strong eventual consistency) — Every replica that has seen the same set of operations shows the same document, no matter what order they arrived in. This is the correctness property the whole design exists to provide.
• Durability (no acknowledged edit lost) — Once the client shows "saved", the operation survives a server crash. This is what pins the write path to a replicated log with a real fsync.
• Concurrent editors (100 per document · 50 typical) — The product limit matters: fan-out per document is bounded, so the hard part is document count, not per-document scale.
• Scale (50 M documents edited per day · 5 M concurrent sessions) — Sessions are long-lived WebSockets, so connection count, not request rate, sizes the fleet.
• Availability (99.99 % for open and edit) — A document that cannot be opened blocks work. Degraded mode is read-only with local edits queued, never a hard error.
• History retention (full history, restorable) — Operations are kept and compacted into snapshots. History is the feature that justifies storing the operation log rather than just the text.

• Operations per second at peak: ~150 k/s — 5 M concurrent sessions, of which maybe 3 % are actively typing at any instant = 150 k typists. Batched at roughly one operation per 100 ms of typing, that is ~150 k ops/s arriving at the servers.
• Fan-out messages per second: ~750 k/s — Each operation is broadcast to the other collaborators in that document. With an average of 6 people in an actively edited document, 150 k × 5 = 750 k pushes/s. This is the number that sizes the connection tier.
• Session servers: ~100–200 — 5 M connections ÷ ~30 k connections per server = 167 servers, and the same fleet must sustain 750 k pushes/s, which at 5 k pushes/s per server would need 150. Both point at the same order, so budget 200 across three availability zones.
• Operation log volume: ~1.3 TB/day — 150 k ops/s × 86 400 s = 13 B ops/day at peak-equivalent, realistically ~5 B/day averaged, at ~150 B each (id, site, position, character, attributes) plus index overhead ≈ ~1 TB/day raw, ~3 TB replicated.
• Document size and snapshot cost: ~50 KB text · ~200 KB with metadata — A 20-page document is ~50 KB of text. A CRDT carries per-character metadata (site id, counter, tombstones) that can be 3–10× the text before compaction, so snapshots are compacted and tombstones garbage collected once every client has acknowledged past them.
• Presence traffic: ~500 k msgs/s — Cursor moves are far more frequent than edits: throttled to 10/s per active user, 50 k moving cursors gives 500 k/s. It is dropped under load, never persisted, and never allowed to queue behind edits, which is why presence runs on its own channel.
• Storage for history: ~1 PB/year — 1 TB/day of operations × 365 = 365 TB/yr raw, ×3 replication ≈ 1.1 PB/yr. Compaction into snapshots plus dropping fine-grained operations older than 30 days (keeping named versions) cuts this by an order of magnitude, which is the actual design decision.
• Reconnect storm after a deploy: ~1.7 k reconnects/s — Draining 200 servers over 10 minutes moves 5 M ÷ 600 s ≈ 8 k connections/s; with jittered backoff spreading each server over 30 s the peak is nearer 1.7 k/s. Each reconnect costs an auth check, a session lookup and a catch-up, so this is what sizes the session store.

• Editor client (local replica + queue) — Holds a full replica of the document and applies every local edit immediately, so typing never waits for the network. Keeps an outbound queue of unacknowledged operations, a vector of what it has seen, and enough state to reconcile after a disconnect. Renders remote cursors from the presence channel, which it treats as disposable.
• WebSocket gateway (sticky · per document) — Terminates the long-lived connection, authenticates the session once at connect, and routes the client to the session server that owns this document. Sticky by document rather than by user, because the whole design depends on one owner per document ordering the operations.
• Session server (one owner per document) — The in-memory authority for a live document: holds the current state, assigns each incoming operation a position in the sequence, rebroadcasts it to the other collaborators, and periodically persists. Single-owner per document is what turns a hard distributed problem into an ordered log with fan-out.
• Ownership registry (lease per document) — Maps document id to the session server that currently owns it, as a short lease refreshed by the owner. A new connection looks here first; if there is no owner, it claims the lease and loads the document. The lease plus a fencing token is what prevents two servers from ordering the same document.
• Operation log (append-only · per document) — The durable record: every acknowledged operation with its sequence number, author and timestamp, partitioned by document id. Append-only writes, range reads by sequence, and a retention policy that compacts old ranges once a snapshot covers them. This log is both the crash-recovery mechanism and the history feature.
• Snapshot store (object storage · compacted state) — Periodic compacted document states, so opening a document is "load the newest snapshot plus the operations after it" instead of replaying a million edits. Snapshots also carry the garbage-collected structure with tombstones removed once every replica has moved past them.
• Document metadata (Postgres · title, owner, ACL) — Title, owner, folder, sharing rules and per-user roles. Small, relational and strongly consistent, because a permission revoke must be immediately effective. Read on open and watched by the session server so a revoke can eject a live collaborator.
• Presence channel (Redis pub/sub · TTL) — Cursor positions, selections and who is in the document, with a few seconds of TTL and no durability. Deliberately separated from the edit path so a burst of cursor movement can be dropped without touching correctness.
• Event bus (Kafka · keyed by doc id) — Carries committed operations and document events to everything that is not the live session: search indexing, notifications, comment anchoring, analytics and export. Keying by document id keeps a document's events ordered for every consumer.
• History service (versions · restore · diff) — Builds the version timeline from the operation log and snapshots, groups operations into human-sized revisions by author and time gap, renders diffs, and performs a restore by appending the inverse operations rather than rewriting history.
• Comments service (anchored to positions) — Comment threads and suggestions, each anchored to a stable range identifier rather than a character offset so concurrent edits do not move it to the wrong text. Handles the case where the anchored text is deleted by marking the comment orphaned instead of losing it.
• Search indexer (debounced per document) — Consumes document events and reindexes the text, debounced so a document being typed into is indexed once every few seconds rather than on every keystroke. Permission fields are indexed alongside so queries filter by access inside the index.
• Notification service (mentions · shares) — Turns document events into notifications: mentions in comments, share invitations, suggestions awaiting review. Batched and deduplicated, because a burst of edits must not become a burst of emails.
• Media storage (images · signed URLs) — Images and attachments, uploaded directly from the client with a pre-signed URL and referenced from the document by id, so the operation log never carries binary payloads and a pasted screenshot does not enter the edit stream.

1. Open a document and join the session — Everything that has to happen before the first keystroke: authorise, find or create the owner, load a state without replaying history, and start receiving other people's edits.
   1. Client opens a connection and presents its token and the document id.
   2. Gateway checks the metadata store for the caller's role on this document.
   3. Gateway asks the ownership registry which session server owns this document.
   4. The owning session server claims or renews the lease with a fencing token.
   5. Session server loads the newest snapshot and replays only the operations after it.
   6. Client receives the state plus the sequence number it is caught up to, and subscribes to presence.
2. Two people type in the same paragraph — The core loop, and the reason this question is hard: both edits apply locally first, arrive in different orders elsewhere, and must still converge to identical text.
   1. Local edit applies to the client replica immediately and renders on the next frame.
   2. Client batches operations from the last few keystrokes and sends them over the socket.
   3. Session server places the operations into the document sequence and resolves them against concurrent edits.
   4. Operations are appended to the durable log before they are acknowledged.
   5. Server broadcasts the operations to every other collaborator and acknowledges the author.
   6. Remote clients apply the operations to their replicas, rebasing their own unacknowledged edits.
   7. The committed operations are published to the bus for everything that is not the live session.
3. Cursors, selections and who is here — Five times the message volume of editing, none of it durable. Kept on its own path so it can be throttled and dropped without ever affecting the document.
   1. Client throttles cursor movement to about ten updates per second.
   2. Presence updates go to the presence channel, not into the operation stream.
   3. The session server subscribes and merges presence into the collaborator list it broadcasts.
   4. Entries carry a TTL so a disappearing client fades out without an explicit goodbye.
   5. Cursor anchors are expressed as character identities, so they survive concurrent edits.
4. Edit offline, then reconnect — The case that rules out any design where the server must see an edit before it exists. Hours of local edits merge without a dialog, and the client catches up on what it missed.
   1. The connection drops; the client keeps applying edits to its local replica and queueing them.
   2. On reconnect, the client presents its last acknowledged sequence number and its queued operations.
   3. Server streams the missed operations; the client applies them beneath its pending edits.
   4. Client sends its queued operations, which the server merges with everything that happened meanwhile.
   5. Operations are appended and broadcast like any other, and the client is marked caught up.
   6. Tombstones and metadata are garbage collected once every live replica has acknowledged past them.
5. Version history, comments and restore — The asynchronous half of the product. Everything here reads the operation log rather than the live session, so none of it can slow down typing.
   1. A background compactor writes a snapshot every few thousand operations.
   2. The history service groups operations into human-sized revisions.
   3. Opening a version renders the document at that point from the nearest snapshot plus a replay.
   4. Restoring a version appends new operations rather than rewriting the log.
   5. Comment anchors are rebased as operations flow past, and orphaned when their text is deleted.
   6. Search reindexes the document a few seconds after typing stops, and mentions become notifications.
   7. Images pasted into the document are uploaded directly to media storage and referenced by id.

Ask: How do concurrent edits to the same region converge, and what does that choice cost?

Good answers name: CRDT with unique character identifiers, Operational transformation with a central server, Locking regions (one editor per paragraph), Last write wins on whole document.

Our pick: A sequence CRDT with unique, totally ordered character identifiers, a single session server per document for ordering, durability and fan-out, and compaction to control the metadata cost.

1. If the CRDT does not need a central server, why have one per document?
   For everything other than convergence: durable ordering into a log, a single place to enforce permissions, fan-out to collaborators without a mesh, snapshotting, and a well-defined point for garbage collection. The CRDT means a brief period with two owners is a performance problem rather than a corruption problem, which is exactly the safety margin you want.
2. How large does the metadata actually get?
   Each character carries an identifier (site id plus counter, around 8–16 bytes) and deleted characters remain as tombstones until collected. A 50 KB document can sit at 200–500 KB before compaction. Mitigations: block-wise identifiers that cover runs of characters rather than one per character, periodic snapshots that renumber, and tombstone collection once every active replica has acknowledged past them.
3. What does interleaving look like, and does it matter?
   Two people typing different words at the same position can produce characters interleaved rather than one word after the other. It is deterministic and identical everywhere, so it is not a correctness bug, and modern algorithms reduce it by keeping runs together. In practice it is rare because collaborators see each other's cursors and do not type in the same spot.
4. How do you test convergence?
   Property-based testing: generate random operation sequences, deliver them to several replicas in different random orders with duplicates, and assert all replicas are byte-identical and that no acknowledged operation is missing. Plus a replay harness over recorded production sessions. This is the test suite that matters most in the whole system.

Ask: How is a document assigned to a server, and what happens when that server dies mid-session?

Good answers name: Lease in a registry plus consistent hashing, fenced by an epoch, Pure consistent hashing with no lease, Any server can serve any document, merging through the CRDT, Consensus group per document.

Our pick: Consistent hashing to pick a candidate, a short lease in the ownership registry to make it authoritative, and a monotonic epoch carried into every log append so a returning old owner is fenced out.

1. What exactly do users see when the owning server crashes?
   Their sockets drop, the editor keeps working locally and queues edits, and reconnection is jittered over a few seconds. A new owner claims the lease once it expires, loads the snapshot plus the tail of the log, and the clients resume from their last acknowledged sequence. The visible effect is a "reconnecting" indicator for a few seconds and no lost work, because unacknowledged edits are still in each client's queue.
2. How do you avoid losing acknowledged operations during that window?
   Acknowledgement happens only after the append to the durable log, so anything acknowledged is recoverable by the new owner. Anything not acknowledged is still in the client queue and is resent on reconnect, where duplicates are harmless because operation identities make replays idempotent.
3. A document has 100 editors and its server is saturated. What do you do?
   Move it: the control path can revoke the lease and place the document on a less loaded server, which is a two-second reconnect for its collaborators. Beyond that, the levers are fan-out helpers that relay broadcasts to a subset of clients, and stricter presence throttling, since presence rather than edits is usually what saturates a big session.
4. How does the gateway know where to route without a registry lookup on every message?
   The lookup happens once at connect; after that the socket is bound to that session server for its lifetime. A lease change forces a reconnect, which is an event rare enough to pay for.

Ask: What is stored: every operation, periodic states, or both — and for how long?

Good answers name: Append-only operation log plus periodic snapshots, Store only the current document state, saved periodically, Log forever, never snapshot, Tiered retention: fine-grained recent, coarse older.

Our pick: An append-only per-document operation log with snapshots every few thousand operations, fine-grained operations retained for 30 days, and daily plus named snapshots kept for the life of the document in cheaper storage.

1. What database would you use for the operation log?
   A wide-column or log-structured store partitioned by document id with the sequence number as the clustering key: appends are sequential, a catch-up is a single range read of one partition, and old ranges are dropped by time bucket. A relational table works up to a point, but per-document partitioning and cheap range deletion are what the workload actually needs.
2. How often do you snapshot, and what decides it?
   Every few thousand operations or a few minutes, whichever comes first, tuned so that the replay after the newest snapshot stays under about 100 ms. Snapshotting is also triggered before the last collaborator leaves, so a dormant document opens instantly next time.
3. How do you delete a document for real when a user asks?
   Tombstone it immediately so it disappears from every surface, then a job removes the operation ranges, snapshots, search index entries, comment threads and media objects, and records the completion. Backups are handled by retention rather than rewriting: the documented window is when the last copy expires, and encrypting each document with its own key lets the key be destroyed immediately.
4. A bug corrupts documents. How do you recover?
   Because the log is append-only, recovery is replay: rebuild each affected document from the last known-good snapshot plus the operations before the bad deploy, and write the result as a new snapshot with a restore entry in history. This is the concrete payoff of never mutating the log in place.

Ask: Where do the 750 k broadcasts per second go, and how do you stop cursors from crowding out edits?

Good answers name: Session server broadcasts directly to its own connections, Broadcast through a pub/sub layer, Separate ephemeral channel for presence, with shedding, Client polling for updates.

Our pick: Direct broadcast from the session server that owns the document, with presence on a separate, throttled, sheddable channel, and batching on both paths so message rate is bounded by documents rather than by keystrokes.

1. What is the first thing to degrade under overload?
   Presence: increase the throttle, coalesce harder, then stop sending cursor updates entirely. Next, increase the edit batching interval from 100 ms to 250 ms, which halves the message rate at a cost users barely notice. Only after that do you reject new sessions on a busy document, and even then existing editors keep working.
2. How do you deploy without disconnecting five million people at once?
   Drain by document: move ownership of documents off a server gradually, with each client receiving a close frame carrying a reconnect delay so the gateway controls the spread. A few percent of documents at a time over ten minutes turns a reconnect storm into a gentle ramp, and the client keeps editing locally throughout.
3. Collaborators are on different continents. What breaks?
   The 200 ms propagation budget, because a round trip to a single owning region can be 150 ms before anything else. Options are to place the document owner near the majority of its collaborators, accept that distant users see more latency, or move to the multi-region merge model, which costs the simple persistence story. The honest answer usually names the first two.
4. How do you measure whether collaboration feels good?
   End-to-end operation latency measured on the client: the time between an edit leaving one client and being applied on another, at p50, p95 and p99, broken down by region. That number, plus the reconnect rate and the share of sessions that ever see a resync, is what the team should page on rather than server CPU.

Ask: How are sharing rules enforced when the document is already open and a revoke happens mid-edit?

Good answers name: Check at connect, then react to change events on the live session, Check permissions on every operation, Short-lived capability tokens refreshed periodically, Enforce in the client.

Our pick: Authorise once at connect, cache the role for the session, subscribe to permission-change events per document, and downgrade or eject the affected socket within a second of a change, with every operation still checked against the cached role's capabilities.

1. What happens to the edits a revoked user made in the gap?
   They stay: they were acknowledged and are part of the document, and removing them would rewrite history. The session is closed, the client is told access was removed, and its queued unsent operations are rejected. If the edits were unwanted, the remedy is the restore feature, which is an ordinary edit.
2. How do commenter and suggester roles work?
   They connect to the same session but with capabilities that allow only comment operations or suggestion operations. A suggestion is modelled as an operation tagged as proposed rather than applied — it lives in the log, it converges like any other operation, and accepting it is a second operation that promotes it into the text.
3. Link sharing is turned off while 50 anonymous viewers are reading. What happens?
   The permission change publishes an event, every session server holding that document re-evaluates its collaborators against the new rule, and sockets that no longer qualify are closed with a reason the editor shows. This is the same code path as a single revoke, applied to a set.
4. How do you keep permission checks off the hot path in search and notifications?
   The search index stores the access list as fields so queries filter inside the index, and it is updated from the same permission-change events. Notifications check permission at send time rather than at enqueue time, because a share can be revoked between the two.