Design Gmail

• Receive mail from any server on the internet: Over SMTP, at the hosts our MX records name. Once we reply 250 to a message we own it: the sender deletes its copy.
• Filter spam, phishing and malware: Cheap, certain checks refuse mail in the SMTP session; uncertain mail is accepted and labelled Spam, where the user can recover it; dangerous attachments are withheld.
• Mailbox with labels and threads: Labels are many to many (Inbox, Starred, Work); archiving is removing the Inbox label. Replies are grouped into conversations.
• Fast inbox on web and mobile, with new mail pushed: First page in a few hundred milliseconds. New mail appears without a refresh, and every device converges on the same labels and read state.
• Search the whole mailbox: Full text plus operators such as from:, has:attachment and before:, across up to 20 years of mail, in under a second.
• Send to anyone: Our own users are delivered directly; other providers over SMTP, with retries, and a bounce to the sender if delivery fails for good. Undo send for a few seconds.
• Attachments: Up to 25 MB per message (Gmail’s public limit; larger files go as links). Scanned for malware, previewed, downloadable from any device.
• IMAP for third-party clients: Labels appear as folders, and changes made in an IMAP client show up on the web and the phone, and the other way round.
• Out of scope: Calendar, chat, contacts beyond autocomplete, ads, smart replies, enterprise admin and legal hold, and training the spam models (we call the model as a scoring service).

• Scale (1.5 B users · 200 B inbound/day): About 2.3 M SMTP messages a second arrive, around 5 M at peak, and three quarters of them are junk. The edge is sized for what arrives; storage and delivery only for the quarter we accept.
• No accepted message is ever lost (durable before 250): Our 250 reply to DATA transfers responsibility from the sender. This drives the hardest trade-off in the design: the 250 must wait for a replicated write, so only checks that take milliseconds can run before it, and everything slow runs after we already own the message.
• Inbox latency (p99 < 300 ms first page): Fifty thread summaries from one range read on one shard. No message bodies and no second shard on this path.
• Search latency (p99 < 1 s over 20 years): Up to a million messages for a heavy user. Only possible if a query touches nothing but that user’s own index.
• Time to the device (p50 < 2 s · p99 < 10 s after 250): Filtering, the mailbox write and the push hint fit in seconds. Attachment sandboxing can take longer, so it holds back the attachment, not the message.
• Spam accuracy (false positives ≪ misses): A real message in Spam is bad; a real message refused is worse, because nobody can recover it. So hard rejects are kept for near-certain signals and uncertain mail is labelled, not refused.
• Availability (reads 99.99 % · inbound may defer): Senders retry a 4xx for days, so a short inbound outage delays mail without losing it, as long as we defer rather than accept into something fragile. A user opening their inbox has no such retry.
• Privacy (owner-only access): Encrypted at rest; no path from one user’s data to another’s through search, caches or shared attachment storage; Trash and Spam purged after 30 days, and deletes completed within a bounded time.

• Inbound SMTP messages: ~2.3 M/s avg · ~5 M/s peak. 200 B/day ÷ 86 400 s ≈ 2.3 M/s. Spam waves and overlapping business hours roughly double it at peak: ~5 M/s. This is what the MX fleet is sized for, though most of it never reaches a disk.
• Accepted deliveries: ~580 k/s · ~33 per user per day. If cheap edge checks refuse about three quarters, 50 B/day are accepted: 50 B ÷ 86 400 ≈ 580 k/s, and 50 B ÷ 1.5 B users ≈ 33 per user per day. About 10 B of those are filed as Spam. Three quarters of the traffic never costs a durable write, which is the whole case for cheap checks first.
• MX servers: ~1,000 at peak. At ~5 k messages/s per server (TLS handshakes and DNS lookups dominate, and most sessions end before DATA), a 5 M/s peak needs 1,000 servers, spread across regions behind our MX records, plus headroom to lose a region.
• New mail stored per day: ~2.5 PB/day · ~0.7 EB/year kept. 50 B × ~50 KB average (most mail is small; attachments carry the bytes) = 2.5 PB/day. Spam expires after 30 days, so ~2 PB/day is kept: about 0.7 EB a year before compression and dedupe.
• Total mailbox storage: ~4.5 EB logical. The quota is 15 GB but the average user holds about 3 GB: 1.5 B × 3 GB = 4.5 EB, roughly 7 EB on disk with erasure coding. About 95 % of it is bodies and attachments that are rarely read after the first week, so they belong in blob storage with cold tiers, not in the mailbox database.
• Metadata per mailbox: ~60 MB average · ~1 GB heaviest. 3 GB ÷ 50 KB ≈ 60 k messages × ~1 KB of headers, labels and index rows ≈ 60 MB, about 90 PB across all users. The heaviest 20-year mailbox has about a million messages, ~1 GB, a small fraction of one shard. No user ever needs to span shards, so partitioning by user always works.
• Search index per user: ~100 MB · ~150 PB total. 60 k messages × ~4 KB of extracted text ≈ 240 MB; a compressed inverted index is roughly 40 % of that, ~100 MB. Across 1.5 B users that is ~150 PB, but a query only ever reads one user’s 100 MB.
• Sync calls: ~700 k/s. ~40 B non-spam deliveries a day ≈ 460 k/s, each nudging ~1.5 devices: up to ~700 k "changes since N" calls a second before coalescing. Each is a range read of a few rows at the head of one user’s history. The busiest read in the system must be the cheapest one.

• Other mail servers (senders and recipients): Every other provider, company server and spammer on the internet. They find our MX hosts in DNS and deliver over SMTP, retrying a 4xx for days; we connect to theirs the same way to send. We control none of them, and a large share of those connecting to us are compromised machines.
• MX servers (stateless SMTP fleet): Terminate SMTP and STARTTLS and run every check that takes milliseconds: IP reputation, rate limits, unknown recipients, SPF, DKIM, DMARC, banned attachment types. Bad mail is refused (5xx) or deferred (4xx) in the session. A message gets 250 only after the inbound queue has stored it durably, so an MX can die at any moment without losing accepted mail, and any MX can take any connection.
• Inbound queue (replicated log · ack before 250): Accepted messages, written to three replicas across zones; an MX waits for two acknowledgements before it replies 250. This is where we take responsibility, and it decouples the edge from filtering and delivery: if delivery slows, mail waits here instead of in senders’ retry queues. Mail our users send to each other enters here too.
• Spam + malware filter (edge features → ML → sandbox): Scores each accepted message in cost order: the features the edge already computed, then content and URL classifiers, then the recipient’s own signals (contacts, filters, past reports), then sandbox detonation for risky attachments. Produces a verdict per recipient: Inbox, Spam or quarantine. Its feedback consumer turns spam-trap hits, campaign clusters and user reports into reputation updates.
• Delivery service (parse · thread · write): Turns a verdict into mailbox state for each recipient: splits the MIME message, stores body and attachments in the blob store, assigns the thread, and writes message, metadata, label index, counters and a history record in one transaction on the recipient’s shard. Idempotent on (user, message hash), because the queue delivers at least once.
• Outbound MTAs (DKIM sign · IP pools): Drain the per-destination queues at each destination’s current rate, resolve its MX, DKIM-sign with our domain key and deliver over TLS from a sending IP chosen by pool. 250 marks the recipient delivered, 4xx reschedules with backoff and slows that destination, 5xx produces a bounce to our user. Normal mail and mail from suspicious accounts use separate IP pools, so one cannot poison the other.
• DNS (MX · SPF · DKIM · DMARC): Publishes our MX records so the world can find us, and every sending domain’s SPF record, DKIM public keys and DMARC policy, which the MX servers query through a caching resolver in each region. Our own deliverability depends on publishing correct SPF, DKIM and DMARC records for our domain here too.
• Reputation store (IP + domain scores in memory): Scores for sending IPs, /24 ranges and domains, built from volume, spam-trap hits, user reports and authentication history. Read on every connection, so it is a replicated in-memory store with sub-millisecond lookups, updated by the filter’s feedback loop within minutes. An unknown IP gets a neutral score and a small rate budget until it earns more.
• Mailbox store (sharded by user_id): Every row for one user lives on one shard: immutable message records, small mutable metadata (labels, read state, thread id), thread summaries, label indexes, unread counters and the history log. Any user operation is a single-shard transaction, and a label change rewrites about 100 bytes, never the message. Shards are replicated groups holding tens of thousands of users each, rebalanced by moving users.
• Blob store (keyed content hash · ref sets): Message bodies and attachments, compressed and encrypted with per-blob keys, addressed by a keyed hash of their content so identical attachments are stored once. Each blob keeps a set of (user, message) references, and a GC deletes it only after that set has been empty for a grace period. Holds about 95 % of all bytes, mostly cold after the first week, so old blobs move to cheaper erasure-coded tiers.
• Outbound queues (per destination domain): One logical queue per destination, grouped by the MX host it resolves to, so a provider that throttles us backs up only its own mail. Each entry is one recipient’s delivery job with its attempt count and next retry time, kept for up to five days. Fed by the send path only after the Sent copy has committed.
• Mail API (REST + IMAP gateway): Authenticates the user and routes every call to the shard that owns their mailbox: list threads by label, read messages, modify labels, search, history since N, send. An IMAP gateway sits beside it and translates folders, UIDs and IDLE onto the same calls. Attachments are authorised through the user’s own message record, never by blob id.
• Search index (per-user inverted index): One small inverted index per user (about 100 MB on average) over sender, recipients, subject, body, attachment names and labels, stored on the same shard as the mailbox. An indexer tails each user’s history and adds new mail within seconds; merges run per user in the background. A query never reads another user’s data, and indexes of users who rarely search stay on cheap disk.
• Push notifier (APNs · FCM · web channel): Watches the shards’ change feed and wakes the user’s devices with { user, history_id }: APNs or FCM for phones, a long-lived channel for open web tabs, IDLE for IMAP. Hints carry no content, are coalesced to at most one per mailbox every few seconds, and may be lost; the history call is the source of truth.
• Web + mobile apps: Keep a local cache of recent threads and bodies keyed by immutable message id, plus the last history id they applied. On a push hint or on reconnect they ask for the changes since that id. Archive, label and read changes are applied locally first and sent as label set operations, which are safe to retry.

1. A message arrives from the internet: The main write path. A sender’s server hands us a message over SMTP; we decide in milliseconds whether to take responsibility for it, then filter, store and index it and tell the user’s phone, all within a few seconds.
   1. The sending server looks up our MX records and connects; the MX checks the connecting IP before it even sends a greeting.
   2. On MAIL FROM and RCPT TO, the MX checks SPF for the envelope domain and that each recipient exists.
   3. After DATA, the MX verifies DKIM and DMARC and the attachment types, appends the message to the inbound queue, and only then replies 250.
   4. The filter scores the message for each recipient and picks a label: Inbox, Spam, or quarantine.
   5. The delivery service stores the body and each attachment in the blob store, keyed by a hash of the content.
   6. It assigns the thread and writes message, labels, thread summary, unread count and a history record in one transaction on the user’s shard.
   7. An indexer tails the user’s history and adds the message to that user’s search index.
   8. The new history id goes out as a hint, and the push notifier wakes the user’s phone and open tabs.
2. Open the inbox, then stay in sync: The main read path. Listing the inbox must be one range read on one shard, and keeping three devices in step must cost a few rows per change, not a re-download.
   1. The app opens and asks for the first page of threads labelled Inbox.
   2. The user opens a thread; the API reads its message records and fetches the bodies from the blob store.
   3. The user archives the thread on their phone.
   4. The change goes out as a hint to the user’s other devices.
   5. The laptop asks for everything that changed since its last history id and applies it.
   6. A tablet that was off for a month sends a history id that is no longer kept, and does a full sync.
3. A botnet doubles inbound traffic: The scale-breaking case for email: a burst that is almost all junk. The design holds because most of it dies before DATA, and nothing slow runs until we have decided to own a message.
   1. Two hundred thousand compromised hosts start connecting, most from IPs we have never seen.
   2. Unknown IPs that exceed their small budget are deferred with 421 and must retry later.
   3. Messages to made-up addresses are refused at RCPT TO, before any body is sent.
   4. Messages forging a brand that publishes DMARC p=reject are refused after DATA.
   5. What survives reaches the queue, and the content classifier files most of it as Spam.
   6. Spam-trap hits and campaign clusters lower the sending IPs’ scores within minutes, and the MX starts refusing them at connect.
4. Send to a provider that is throttling us: The send path and its failure mode. Once the user sees "Sent" we owe them delivery or a bounce, while the receiving provider answers 421 and the retry clock runs for days.
   1. The user sends a message with an attachment to a friend at another provider and a colleague on our service.
   2. When the undo window passes, the colleague’s copy goes to the inbound queue and the friend’s to the outbound queue for their domain.
   3. An outbound MTA takes the job at that destination’s current rate, signs it with DKIM and connects to the domain’s MX.
   4. The receiver answers 421: it is rate limiting us. The MTA reschedules the job and slows that destination down.
   5. A later attempt gets 250 and the job is done; had it still been failing after five days, or got a 5xx, the MTA would bounce it to the sender.
5. Search twenty years of mail: Search is per user by construction: a query never leaves one user’s index, which is why it can be fast over 20 years and safe without a user filter on every posting.
   1. The user opens the app, and the API warms that user’s search index while they read.
   2. The user searches for invoices from Alice with an attachment, before 2019.
   3. The index intersects the posting lists for those terms inside this user’s index.
   4. Messages newer than the index are scanned directly from the mailbox and merged in.
   5. The API turns the hits into thread summaries, drops anything deleted since, and returns the first page.
   6. In the background, the user’s delta segments are merged and deleted messages are purged from the index.

Ask: A message looks suspicious. Do you refuse it in the SMTP session, accept it and file it as spam, or accept it and bounce it later?

Good answers name: Refuse in the session on cheap, certain signals; accept the rest and label uncertain mail Spam, Accept everything, decide later, bounce what is bad, Accept and silently drop what looks bad, Run the full content model and sandbox before replying.

Our pick: The 250 is our commit point, so it waits for the message to be appended to the replicated inbound queue (two of three replicas, in different zones), and an MX holds nothing on its own disk that matters. Before the 250 run only checks that take milliseconds and are near certain: IP and range reputation from an in-memory store, per-IP rate and connection limits, recipient existence, SPF, DKIM and DMARC (refusing where the domain publishes p=reject), size limits, banned attachment types and antivirus signature hits. Unknown or bursting senders get 421 rather than 5xx, because deferral is free for a real server and expensive for a botnet. Everything else runs after the 250 and produces a label, never a bounce: Inbox, Spam, or a quarantine that withholds a dangerous attachment. Our own bounces are reserved for real delivery failures, never for spam verdicts. Google’s public guidance for bulk senders has the same shape: unauthenticated mail and excessive rates are refused or deferred in the SMTP session, while content judgements land in the Spam folder.

1. The MX crashes after the queue append but before the sender receives 250. What happens?
   The sender never saw 250, so it keeps the message and retries, and we receive it twice. This is SMTP’s classic duplicate, and it is the right way to fail: a duplicate is recoverable, a loss is not. Delivery dedupes on (recipient, Message-ID, body hash) for a few days, so the user sees one copy. A message without a Message-ID gets one at the edge and can only be deduped by body hash.
2. The inbound queue is slow or down. What does the edge do?
   Stop answering 250 and answer 421 instead, so senders keep the mail and retry. That turns our outage into a delay, which SMTP is built for: senders retry for days. What we must never do is fall back to accepting onto an MX’s local disk without replication, because then an MX failure loses mail we promised to deliver. Queue capacity per region is alarmed long before this point.
3. Forwarding and mailing lists break SPF and sometimes DKIM. Doesn’t strict DMARC refuse legitimate mail?
   Yes, that is the known weak spot. A forwarder sends from its own IP, so SPF fails for the original domain, and a list that adds a footer or tags the subject breaks DKIM. Mitigations: honour ARC (RFC 8617), where a trusted intermediary records the authentication results it saw; treat DMARC failures from known forwarders and large lists as a spam signal rather than a refusal; and recognise users’ own forwarding setups. Lists that rewrite From to their own domain avoid the problem.
4. Why not greylist everything, deferring the first message from any new sender?
   It beats simple bots, but it delays every first email from a new correspondent by the sender’s retry interval, often 15 to 30 minutes, which users hate for sign-up codes and password resets. Serious spam infrastructure retries correctly, so the benefit has shrunk. I would defer only unknown IPs that exceed a small budget, not every first contact.
5. How do you know the edge is refusing the right things?
   Sample refused sessions and replay them through the full classifier offline: the share it calls legitimate is the edge’s false-reject rate, and it must be tiny. Track refuse and defer rates by reason, by sending provider and by region, and alert on sudden changes: a spike in DMARC refusals for one big sender usually means they broke their DKIM, not that they turned to spam. Postmaster feedback to large senders closes the loop.

Ask: How do you store 90 trillion messages so the inbox is one range read and a label change is a tiny write?

Good answers name: Shard by user id: immutable message records, small mutable metadata, bodies in blob storage, Mailbox files (Maildir or mbox) on file servers, One global message table partitioned by message id, Labels and flags stored inside the message record.

Our pick: A sharded store whose shard key is user_id, with every row for a user on one shard: sharded MySQL with Vitess, or a Spanner-class database with the user as the root of interleaved tables. What matters is single-shard transactions, not the brand. Tables: messages (user, msg_id) with parsed headers, size, snippet and blob references, written once; meta (user, msg_id) with labels, read and starred flags and thread_id; threads (user, thread_id) with the summary the list shows; label_index (user, label, last_ts desc, thread_id); counters (user, label); history (user, history_id). A delivery or a label change updates its rows and appends one history record in one transaction. Thread ids are assigned at delivery from In-Reply-To and References against the user’s Message-ID index, falling back to the normalised subject with an overlapping participant in a recent window. Each shard is a replicated group of tens of thousands of users across zones; a growing shard is split by moving users. Bodies and attachments go to the blob store with the other 95 % of the bytes.

1. How exactly do you pick the thread for a new message?
   Look up its In-Reply-To and References ids in this user’s (user, Message-ID) → thread index; the first hit wins. With no hit, normalise the subject (strip Re:, Fwd:, AW: and list tags like [team]) and look for a thread with the same normalised subject and an overlapping participant in the last few weeks; subject alone would merge every "Invoice" ever received. When a reply arrives before its parent, it starts a thread and records each id it references, so the parent finds that thread when it lands. A changed subject starts a new thread, and very long threads are capped (say at 100 messages) so a thread stays bounded.
2. A shard holding 30,000 users loses its leader. What do those users see?
   Shards are replicated groups with consensus across three zones, so a leader election takes a few seconds, during which those users’ writes wait and reads can be served slightly stale from a replica. Their inbound mail waits in the inbound queue, which is exactly why the queue exists. Because no user spans shards, a failure affects a known set of users completely and everyone else not at all.
3. Why not Cassandra with the user as the partition key?
   It fits the access pattern: partition by user, cluster by label and time. The gap is transactions: a delivery writes the message, label index, counter and history record together, and history ids must increase without races. Cassandra gives that only through lightweight transactions on every write, which are slow, or through idempotent rows written in a careful order with the history record last. It can be made to work; a store with single-shard transactions makes it much easier to be correct.
4. How do unread counts stay right across 1.5 B mailboxes?
   They are counters updated in the same transaction as each label or read-state change, so they are exact within a shard by construction. Bugs still happen, so a background job recomputes counts from label_index for a sample of users, and for any user who reports a wrong badge, and repairs drift. The badge reads the counter, never a count query.
5. One account receives 2 million messages a day, say a big company’s catch-all address. Does partitioning by user still work?
   It strains it: 2 M a day is about 23 writes a second sustained, which one shard handles, but the mailbox grows by tens of gigabytes a month and blows through any quota. This is a product limit as much as a storage one: per-account daily receive limits, a quota that rejects with 452 or 552 once full, and steering such workloads to a business product with retention policies. The design should not be bent for the one case the quota exists to prevent.

Ask: The same 2 MB PDF arrives for a million users. Do you store it once, and what does sharing one copy across users cost you in privacy?

Good answers name: Server-side dedupe across all users by keyed content hash, with reference sets, Per-user dedupe only, Client-side dedupe: upload only if the server lacks this hash, No dedupe.

Our pick: Split the MIME message at delivery: the text and HTML bodies go into one compressed object per message, and each attachment above ~64 KB becomes a blob whose id is HMAC-SHA256 of its bytes under a service secret, so a blob id that leaks into a log cannot be matched against a known file outside our systems. Each blob keeps a reference set of (user, msg_id) rows; delivery adds its reference before the mailbox transaction commits, and deletion removes it after the message is purged, so any failure leaves an extra reference (a leak the GC finds later) and never a missing one. A blob whose set is empty is deleted only after a grace period of days and a re-check. Every blob is encrypted with its own random data key, wrapped by a key-management service. Clients never see blob ids: an attachment is fetched as (message, part) and authorised through the user’s own message record. Quota charges each user the full size, and tenants that hold their own keys get their own dedupe domain.

1. Why a reference set instead of a counter?
   Delivery is at least once. A retry that increments a counter twice leaves a count that never reaches zero, a lost decrement leaves it stuck, and you cannot tell which references are real. With a set of (user, msg_id), adding the same reference twice is a no-op, and the GC can verify every entry against the mailbox it names. For a blob with millions of references the set is stored as its own rows under the blob id, so it is a range, not one giant value.
2. A user deletes their account. Is their attachment gone if 10,000 other people received the same file?
   Their copy is: their messages and references are removed, and nothing is reachable from their account. The bytes remain only because 10,000 other people independently received the same file, and those copies are theirs, exactly as if each had been stored separately. When the last reference goes, the blob is deleted after the grace period.
3. Could someone learn that another user has a file by sending it to themselves and timing delivery?
   Not in this design. The sender always transmits every byte, delivery does the same work whether the blob is new or already stored, and no response reveals which it was. If the store wrote measurably faster on a hit, we would equalise that path. The attack that hurt file-sync products needed the client to skip the upload, which email never lets it do.
4. A 25 MB attachment is sent in one message to 500 of our users. What happens?
   The MX receives it once, the queue stores it once, and delivery writes one blob and adds 500 references, one in each recipient’s transaction: 500 small mailbox writes and one 25 MB blob write instead of 12.5 GB. Each user is still charged 25 MB of quota, because they think of it as their own copy and our saving should not change what they see.
5. Where does malware scanning fit with dedupe?
   Scan results attach to the blob, so a file sent to a million users is detonated once. If a new signature later flags it, every message referencing it can have the attachment withheld at read time without touching a million mailboxes. Sharing one copy is a real security benefit here.

Ask: Why not put all mail in one big search cluster, the way web search does?

Good answers name: A per-user inverted index stored with the mailbox and updated from its history, A shared search cluster routed by user id (Elasticsearch with routing), One global index over all mail, No index: scan the mailbox at query time.

Our pick: Each user has an inverted index on the same shard as their mailbox, with fields for from, to, cc, subject, body text, attachment names and labels, plus terms for operators such as has:attachment and is:unread. Documents are numbered in date order per user, so posting lists are date-sorted and the default newest-first query stops at the first page. An indexer co-located with the shard tails each user’s history: new messages go into a small delta segment, label changes update the label terms, and a background merge per user keeps the index to a few segments and purges deletes. A query reads the index plus a direct scan of messages newer than the index’s last applied history id, so mail is searchable the moment it is in the inbox. Indexes of users who have not searched for weeks live on cheaper disk and are prefetched when the app opens. The scan-the-mailbox alternative survives only in that small role: covering the few messages the indexer has not reached.

1. A user with a million-message mailbox searches for a word that matches most of it. What happens?
   The query only needs the newest 50 hits, and postings are in date order, so the engine walks from the newest end and stops after 50: cost is bounded by the page size, not by the match count. A result count like "about 120,000" is an estimate from the posting list length, not computed exactly.
2. Labels change constantly. Do label changes rewrite the index?
   No. Text never changes after delivery, so the large structures are append-only per user; labels live in a small separate structure per user, updated in place from history, and are re-checked against current metadata when hits are hydrated. A search for label:inbox therefore reflects an archive within a second.
3. A bug corrupts some users’ indexes. How do you recover?
   The index is derived from the mailbox, so a job rebuilds an affected user’s index from their messages, swaps it in, and replays any history since the rebuild started. Because indexes are per user, the repair is small and local. A fleet-wide format change works the same way, shard by shard in the background, serving the old format until each user’s new index is ready.
4. Do you index the contents of attachments?
   Asynchronously and selectively. Extracting text from PDFs and documents is expensive, attachments are shared across users, and most are never searched. Extract once per blob (dedupe helps again), store the text with the blob, and add it to each referencing user’s index lazily; attachment names are indexed at delivery because that is what people mostly search for.
5. How would you measure search quality, not just its speed?
   Log which result the user opened and at what rank, and how often a search is refined or abandoned. The success rate is the share of searches where the user opens a result from the first page; a drop after a change to parsing or ranking is the alarm. Because the data is private, offline evaluation uses consenting test accounts and synthetic mailboxes, not real users’ mail.

Ask: A phone, a laptop and an IMAP client show the same mailbox. How does each one find out what changed, and what happens when a notification is lost?

Good answers name: Per-mailbox history id with a change log, plus content-free push hints, Changed-since timestamps, Full diff: the client sends what it has, the server compares, Push the changes themselves to devices.

Our pick: Every change to a mailbox (message added, labels added or removed, message deleted) appends a history record with that user’s next history id, in the same transaction as the change. The API serves "changes since N" as a range read; records are kept for at least a week, and a client with an older id gets 404 and does a full sync (re-list recent threads per label, reconcile by message id, adopt the current id), which is the same contract the public Gmail API documents. Hints carry only { user, history_id }: APNs or FCM for phones, a long-lived channel for web tabs, IDLE for IMAP clients. Client changes are label set operations (add Work, remove INBOX), so retrying them is harmless, and when two devices touch the same label the server’s history order decides. IMAP is a translation layer: each label is a folder with its own UIDs, history records become EXISTS and EXPUNGE responses, and CONDSTORE mod-sequences map onto history ids.

1. The phone archives a thread offline while the laptop labels it Work. Who wins?
   Both, because they touch different labels: the phone removes INBOX, the laptop adds Work, and set operations on different elements commute. When both touch the same label (one archives, the other moves it back to the inbox), the server applies them in arrival order and both devices converge on its state through history. Last writer wins is acceptable for labels; it would not be for new mail, and new mail cannot be lost this way because only the server adds messages.
2. Why must the history id be assigned inside the same transaction as the change?
   Otherwise a client can read history up to N, and then a change commits with an id below N that it will never ask for. Assigning the id inside the transaction on the user’s shard makes ids appear in commit order, so "everything after N" is complete. Ids may have gaps (the public Gmail API says its ids increase but are not contiguous), which is fine: completeness needs order, not density.
3. A busy user has 40 changes a minute and five devices. Isn’t that a lot of pushes?
   The notifier coalesces per mailbox, at most one hint every few seconds, carrying the latest id. Foreground devices hold a live channel and fetch at once; a backgrounded phone gets a push only for mail that should notify the user, and syncs the rest when opened. The work per change stays a read of a few rows at the head of one user’s history.
4. IMAP clients expect folders with stable UIDs. How do labels map without breaking them?
   Each label is exposed as a folder, and the gateway keeps a UID sequence per folder: gaining a label gives the message a new UID in that folder, losing it is an EXPUNGE there. Moving a message between folders becomes adding one label and removing another, and deleting from a label folder removes that label rather than the message. Gmail’s own IMAP service documents a similar mapping, with an All Mail folder holding everything.
5. How would you know sync is broken for some users?
   Canary accounts on every client version make changes and measure how long other devices take to reflect them. Track the rate of 404 full syncs, since a spike means history is trimmed too aggressively or clients are stuck, and have a sample of clients report a checksum of their cached labels for recent threads to compare with the server’s.

Ask: Yahoo starts answering 421 to a third of your mail. How do you keep delivering to them, and to everyone else, without getting blocklisted?

Good answers name: Per-destination queues with adaptive rates, backoff on 4xx, bounce on 5xx, separate IP pools, One shared FIFO queue with retries, Relay through a commercial email provider, Send synchronously from the API while the user waits.

Our pick: After the Sent copy commits, each external recipient becomes a job in the queue for its destination, grouped by the MX host the domain resolves to, so all domains hosted by one provider share one politeness budget. Outbound MTAs take jobs at each destination’s current rate, which starts from history, halves on a burst of 4xx and grows slowly while 250s come back; tokens live in a shared limiter and MTAs lease them in small batches. A 4xx reschedules the job with exponential backoff and jitter for up to five days, with a "still trying" notice to the sender after a few hours; a 5xx, or the five-day limit, produces a bounce to our own user. Every message is DKIM-signed with d=ourmail.com and a rotating selector, and our SPF and DMARC records cover the sending IPs. Outbound mail is scored like inbound: accounts that look compromised are rate limited and sent from a separate IP pool, and new IPs are warmed up over weeks. Receivers’ complaint feedback loops and postmaster dashboards are a primary signal, not an afterthought.

1. An account is compromised and sends 10,000 spam messages in an hour. What stops it?
   Per-account limits first: a daily recipient cap and a rate limit that is tighter for new or unusual senders. The outbound filter scores content with the same models as inbound, so a burst of near-identical messages to strangers scores high, is held or sent from the risky pool, and the account is challenged to re-authenticate or suspended. The main pool’s reputation never sees most of it.
2. The remote server answers 250, then later sends a bounce anyway. What do you do?
   That is an asynchronous bounce: the receiver accepted, then failed internally. It arrives as an inbound delivery status notification to our envelope sender, so we use a VERP-style envelope sender (bounce+<message id>@ourmail.com) to match it to the original message and recipient. It is delivered to the user like any bounce, and repeated hard bounces for an address feed a warning the next time they write to it.
3. How do 1,000 MTAs share one rate limit for Yahoo?
   A shared limiter holds a token bucket per destination, replicated in memory, and each MTA leases a small batch of tokens at a time, so no MTA makes a round trip per message. A 4xx seen by any MTA lowers the shared rate. If the limiter is unavailable, MTAs fall back to a small fixed local share rather than sending unthrottled.
4. How do you start sending from a brand-new IP range?
   Warm it up: a small daily volume to each large provider at first, choosing the mail most likely to be wanted (replies to existing threads, mail to frequent contacts), doubling over days to weeks while bounce and complaint rates stay low. Receivers judge an IP by its history, and a new IP that suddenly sends a flood looks exactly like a spammer.
5. What should "delivered" mean in the UI?
   For external mail, nothing more than sent: an SMTP 250 means the receiving server took responsibility, not that the recipient read it or that it reached their inbox rather than their spam folder. We show the message as sent and report failures, the bounces, explicitly. Claiming more would be inventing information we do not have.