Design Gmail
Receive, filter, store and search email for 1.5 billion users: 200 billion inbound messages a day, mostly spam, kept for 20 years.
Last updated 2026-10-26. Difficulty: hard. Patterns: smtp-ingress, spam-filtering, per-user-sharding, inverted-index, sync. Reported at Google, Microsoft, Amazon, Meta, Apple, Dropbox.
The interviewer asks, the candidate answers and draws, and you press Next. Skip to any stage or key moment, and ask the AI Mentor anything along the way.
Or sit it in the style of Google · Microsoft · Amazon · Meta, run as that company’s round is reported to go.
The short answer
How to design Gmail in a system design interview, in five steps:
- Scope it. Receive mail from any server on the internet, filter spam, phishing and malware, mailbox with labels and threads, fast inbox on web and mobile, with new mail pushed, search the whole mailbox, send to anyone, attachments, IMAP for third-party clients.
- Set the targets. Scale: 1.5 B users · 200 B inbound/day; No accepted message is ever lost: durable before 250; Inbox latency: p99 < 300 ms first page; Search latency: p99 < 1 s over 20 years.
- Do the numbers. Inbound SMTP messages: ~2.3 M/s avg · ~5 M/s peak; Accepted deliveries: ~580 k/s · ~33 per user per day; MX servers: ~1,000 at peak.
- Draw the design. Other mail servers, MX servers, Inbound queue, Spam + malware filter, Delivery service, Outbound MTAs, DNS, Reputation store, Mailbox store.
- Go deep where it is hard.
- Owning a message: reject, defer or accept → Refuse in the session on cheap, certain signals; accept the rest and label uncertain mail Spam.
- The mailbox store and its shard key → Shard by user id: immutable message records, small mutable metadata, bodies in blob storage.
- Attachments: dedupe without leaking → Server-side dedupe across all users by keyed content hash, with reference sets.
- Search: one index per user → A per-user inverted index stored with the mailbox and updated from its history.
- Keeping every device in step → Per-mailbox history id with a change log, plus content-free push hints.
- Sending without getting blocked → Per-destination queues with adaptive rates, backoff on 4xx, bounce on 5xx, separate IP pools.
Functional requirements
- Receive mail from any server on the internet. Over SMTP, at the hosts our MX records name. Once we reply 250 to a message we own it: the sender deletes its copy.
- Filter spam, phishing and malware. Cheap, certain checks refuse mail in the SMTP session; uncertain mail is accepted and labelled Spam, where the user can recover it; dangerous attachments are withheld.
- Mailbox with labels and threads. Labels are many to many (Inbox, Starred, Work); archiving is removing the Inbox label. Replies are grouped into conversations.
- Fast inbox on web and mobile, with new mail pushed. First page in a few hundred milliseconds. New mail appears without a refresh, and every device converges on the same labels and read state.
- Search the whole mailbox. Full text plus operators such as from:, has:attachment and before:, across up to 20 years of mail, in under a second.
- Send to anyone. Our own users are delivered directly; other providers over SMTP, with retries, and a bounce to the sender if delivery fails for good. Undo send for a few seconds.
- Attachments. Up to 25 MB per message (Gmail’s public limit; larger files go as links). Scanned for malware, previewed, downloadable from any device.
- IMAP for third-party clients. Labels appear as folders, and changes made in an IMAP client show up on the web and the phone, and the other way round.
- Out of scope. Calendar, chat, contacts beyond autocomplete, ads, smart replies, enterprise admin and legal hold, and training the spam models (we call the model as a scoring service).
Non-functional requirements
- Scale (1.5 B users · 200 B inbound/day). About 2.3 M SMTP messages a second arrive, around 5 M at peak, and three quarters of them are junk. The edge is sized for what arrives; storage and delivery only for the quarter we accept.
- No accepted message is ever lost (durable before 250). Our 250 reply to DATA transfers responsibility from the sender. This drives the hardest trade-off in the design: the 250 must wait for a replicated write, so only checks that take milliseconds can run before it, and everything slow runs after we already own the message.
- Inbox latency (p99 < 300 ms first page). Fifty thread summaries from one range read on one shard. No message bodies and no second shard on this path.
- Search latency (p99 < 1 s over 20 years). Up to a million messages for a heavy user. Only possible if a query touches nothing but that user’s own index.
- Time to the device (p50 < 2 s · p99 < 10 s after 250). Filtering, the mailbox write and the push hint fit in seconds. Attachment sandboxing can take longer, so it holds back the attachment, not the message.
- Spam accuracy (false positives ≪ misses). A real message in Spam is bad; a real message refused is worse, because nobody can recover it. So hard rejects are kept for near-certain signals and uncertain mail is labelled, not refused.
- Availability (reads 99.99 % · inbound may defer). Senders retry a 4xx for days, so a short inbound outage delays mail without losing it, as long as we defer rather than accept into something fragile. A user opening their inbox has no such retry.
- Privacy (owner-only access). Encrypted at rest; no path from one user’s data to another’s through search, caches or shared attachment storage; Trash and Spam purged after 30 days, and deletes completed within a bounded time.
Back-of-envelope estimates
- Inbound SMTP messages: ~2.3 M/s avg · ~5 M/s peak. 200 B/day ÷ 86 400 s ≈ 2.3 M/s. Spam waves and overlapping business hours roughly double it at peak: ~5 M/s. This is what the MX fleet is sized for, though most of it never reaches a disk.
- Accepted deliveries: ~580 k/s · ~33 per user per day. If cheap edge checks refuse about three quarters, 50 B/day are accepted: 50 B ÷ 86 400 ≈ 580 k/s, and 50 B ÷ 1.5 B users ≈ 33 per user per day. About 10 B of those are filed as Spam. Three quarters of the traffic never costs a durable write, which is the whole case for cheap checks first.
- MX servers: ~1,000 at peak. At ~5 k messages/s per server (TLS handshakes and DNS lookups dominate, and most sessions end before DATA), a 5 M/s peak needs 1,000 servers, spread across regions behind our MX records, plus headroom to lose a region.
- New mail stored per day: ~2.5 PB/day · ~0.7 EB/year kept. 50 B × ~50 KB average (most mail is small; attachments carry the bytes) = 2.5 PB/day. Spam expires after 30 days, so ~2 PB/day is kept: about 0.7 EB a year before compression and dedupe.
- Total mailbox storage: ~4.5 EB logical. The quota is 15 GB but the average user holds about 3 GB: 1.5 B × 3 GB = 4.5 EB, roughly 7 EB on disk with erasure coding. About 95 % of it is bodies and attachments that are rarely read after the first week, so they belong in blob storage with cold tiers, not in the mailbox database.
- Metadata per mailbox: ~60 MB average · ~1 GB heaviest. 3 GB ÷ 50 KB ≈ 60 k messages × ~1 KB of headers, labels and index rows ≈ 60 MB, about 90 PB across all users. The heaviest 20-year mailbox has about a million messages, ~1 GB, a small fraction of one shard. No user ever needs to span shards, so partitioning by user always works.
- Search index per user: ~100 MB · ~150 PB total. 60 k messages × ~4 KB of extracted text ≈ 240 MB; a compressed inverted index is roughly 40 % of that, ~100 MB. Across 1.5 B users that is ~150 PB, but a query only ever reads one user’s 100 MB.
- Sync calls: ~700 k/s. ~40 B non-spam deliveries a day ≈ 460 k/s, each nudging ~1.5 devices: up to ~700 k "changes since N" calls a second before coalescing. Each is a range read of a few rows at the head of one user’s history. The busiest read in the system must be the cheapest one.
Components
- Other mail servers (senders and recipients): Every other provider, company server and spammer on the internet. They find our MX hosts in DNS and deliver over SMTP, retrying a 4xx for days; we connect to theirs the same way to send. We control none of them, and a large share of those connecting to us are compromised machines.
- MX servers (stateless SMTP fleet): Terminate SMTP and STARTTLS and run every check that takes milliseconds: IP reputation, rate limits, unknown recipients, SPF, DKIM, DMARC, banned attachment types. Bad mail is refused (5xx) or deferred (4xx) in the session. A message gets 250 only after the inbound queue has stored it durably, so an MX can die at any moment without losing accepted mail, and any MX can take any connection.
- Inbound queue (replicated log · ack before 250): Accepted messages, written to three replicas across zones; an MX waits for two acknowledgements before it replies 250. This is where we take responsibility, and it decouples the edge from filtering and delivery: if delivery slows, mail waits here instead of in senders’ retry queues. Mail our users send to each other enters here too.
- Spam + malware filter (edge features → ML → sandbox): Scores each accepted message in cost order: the features the edge already computed, then content and URL classifiers, then the recipient’s own signals (contacts, filters, past reports), then sandbox detonation for risky attachments. Produces a verdict per recipient: Inbox, Spam or quarantine. Its feedback consumer turns spam-trap hits, campaign clusters and user reports into reputation updates.
- Delivery service (parse · thread · write): Turns a verdict into mailbox state for each recipient: splits the MIME message, stores body and attachments in the blob store, assigns the thread, and writes message, metadata, label index, counters and a history record in one transaction on the recipient’s shard. Idempotent on (user, message hash), because the queue delivers at least once.
- Outbound MTAs (DKIM sign · IP pools): Drain the per-destination queues at each destination’s current rate, resolve its MX, DKIM-sign with our domain key and deliver over TLS from a sending IP chosen by pool. 250 marks the recipient delivered, 4xx reschedules with backoff and slows that destination, 5xx produces a bounce to our user. Normal mail and mail from suspicious accounts use separate IP pools, so one cannot poison the other.
- DNS (MX · SPF · DKIM · DMARC): Publishes our MX records so the world can find us, and every sending domain’s SPF record, DKIM public keys and DMARC policy, which the MX servers query through a caching resolver in each region. Our own deliverability depends on publishing correct SPF, DKIM and DMARC records for our domain here too.
- Reputation store (IP + domain scores in memory): Scores for sending IPs, /24 ranges and domains, built from volume, spam-trap hits, user reports and authentication history. Read on every connection, so it is a replicated in-memory store with sub-millisecond lookups, updated by the filter’s feedback loop within minutes. An unknown IP gets a neutral score and a small rate budget until it earns more.
- Mailbox store (sharded by user_id): Every row for one user lives on one shard: immutable message records, small mutable metadata (labels, read state, thread id), thread summaries, label indexes, unread counters and the history log. Any user operation is a single-shard transaction, and a label change rewrites about 100 bytes, never the message. Shards are replicated groups holding tens of thousands of users each, rebalanced by moving users.
- Blob store (keyed content hash · ref sets): Message bodies and attachments, compressed and encrypted with per-blob keys, addressed by a keyed hash of their content so identical attachments are stored once. Each blob keeps a set of (user, message) references, and a GC deletes it only after that set has been empty for a grace period. Holds about 95 % of all bytes, mostly cold after the first week, so old blobs move to cheaper erasure-coded tiers.
- Outbound queues (per destination domain): One logical queue per destination, grouped by the MX host it resolves to, so a provider that throttles us backs up only its own mail. Each entry is one recipient’s delivery job with its attempt count and next retry time, kept for up to five days. Fed by the send path only after the Sent copy has committed.
- Mail API (REST + IMAP gateway): Authenticates the user and routes every call to the shard that owns their mailbox: list threads by label, read messages, modify labels, search, history since N, send. An IMAP gateway sits beside it and translates folders, UIDs and IDLE onto the same calls. Attachments are authorised through the user’s own message record, never by blob id.
- Search index (per-user inverted index): One small inverted index per user (about 100 MB on average) over sender, recipients, subject, body, attachment names and labels, stored on the same shard as the mailbox. An indexer tails each user’s history and adds new mail within seconds; merges run per user in the background. A query never reads another user’s data, and indexes of users who rarely search stay on cheap disk.
- Push notifier (APNs · FCM · web channel): Watches the shards’ change feed and wakes the user’s devices with { user, history_id }: APNs or FCM for phones, a long-lived channel for open web tabs, IDLE for IMAP. Hints carry no content, are coalesced to at most one per mailbox every few seconds, and may be lost; the history call is the source of truth.
- Web + mobile apps: Keep a local cache of recent threads and bodies keyed by immutable message id, plus the last history id they applied. On a push hint or on reconnect they ask for the changes since that id. Archive, label and read changes are applied locally first and sent as label set operations, which are safe to retry.
User flows
- A message arrives from the internet. The main write path. A sender’s server hands us a message over SMTP; we decide in milliseconds whether to take responsibility for it, then filter, store and index it and tell the user’s phone, all within a few seconds.
- The sending server looks up our MX records and connects; the MX checks the connecting IP before it even sends a greeting. This is the cheapest place to stop abuse: a known-bad IP gets 554 instead of a greeting and we never see a byte of its mail. An unknown IP gets a neutral score and a small per-IP budget, so a fresh botnet cannot pour millions of messages in before reputation catches up. The lookup is an in-memory read of well under a millisecond.
- On MAIL FROM and RCPT TO, the MX checks SPF for the envelope domain and that each recipient exists. SPF asks whether this IP may send for the envelope sender’s domain, from a TXT record. Unknown recipients get 550 5.1.1 before DATA, which removes dictionary attacks without ever receiving their bodies. An SPF failure alone is only a signal, because forwarding breaks SPF for legitimate mail; DMARC decides what it means.
- After DATA, the MX verifies DKIM and DMARC and the attachment types, appends the message to the inbound queue, and only then replies 250. DKIM proves the signing domain vouched for these exact headers and body; DMARC says what the From domain wants done when neither SPF nor DKIM aligns with it, and p=reject lets us refuse in the session. The append waits for two of three replicas in different zones; only then does 250 move responsibility from the sender to us. If the MX dies before the 250, the sender still has the message and retries.
- The filter scores the message for each recipient and picks a label: Inbox, Spam, or quarantine. The cheap features came with the message from the edge, so content and URL classifiers run next, in tens of milliseconds. The recipient’s own signals change the answer: a sender in their contacts, or one they reported last week. Risky attachments (macros, archives) go to a sandbox that can take seconds; the message is delivered meanwhile with the attachment held until it clears.
- The delivery service stores the body and each attachment in the blob store, keyed by a hash of the content. A 2 MB PDF that is already stored (a newsletter sent separately to a million of our users) costs one more reference, not another 2 MB. The reference names this user and message, so adding it again on a retry is harmless. The MIME skeleton is kept too, so the original bytes can be rebuilt exactly for "show original".
- It assigns the thread and writes message, labels, thread summary, unread count and a history record in one transaction on the user’s shard. The thread comes from In-Reply-To and References, looked up in this user’s Message-ID index; with no match, the normalised subject plus an overlapping participant in recent weeks. The history id is the next number in this user’s sequence, written in the same transaction, so a client sees the message and its change record together or not at all. The write is idempotent on (user, message hash), because the queue redelivers after crashes.
- An indexer tails the user’s history and adds the message to that user’s search index. Indexing is off the delivery path, so a slow indexer delays search by seconds rather than delaying mail. New postings go into a small per-user delta segment that is merged later. Searches also scan the few messages newer than the index, so a message is findable as soon as it is in the inbox.
- The new history id goes out as a hint, and the push notifier wakes the user’s phone and open tabs. The hint carries no content, only the user and the new history id. A lost or duplicated hint costs nothing, because the device then asks what changed since its own last history id and builds the notification from that. Mail labelled Spam produces no notification at all.
- Open the inbox, then stay in sync. The main read path. Listing the inbox must be one range read on one shard, and keeping three devices in step must cost a few rows per change, not a re-download.
- The app opens and asks for the first page of threads labelled Inbox. label_index is ordered by (user, label, last message time descending, thread id), so this is a range read of 50 rows on one shard, a few milliseconds. Thread summaries carry senders, snippet, unread count and labels, so no bodies are read. The response includes the mailbox’s current history id, which the client keeps.
- The user opens a thread; the API reads its message records and fetches the bodies from the blob store. Bodies are immutable, so the client caches them forever by message id and never asks twice. Attachments are fetched only when opened, and access is checked through this user’s message record, never by blob id, so sharing one stored copy with other users opens no path to theirs.
- The user archives the thread on their phone. Archive is "remove the Inbox label" from each message in the thread: a few hundred bytes across metadata, label index and counters, plus one history record, in one transaction. The messages themselves are untouched. Label changes are set operations, so a retried or duplicated request gives the same result.
- The change goes out as a hint to the user’s other devices. Every committed change, whether a delivery or an archive from another device, puts { user, history_id } on the shard’s change feed. The notifier coalesces per mailbox to at most one hint every few seconds, so archiving 30 threads in a row is one push. The device that made the change ignores a hint for an id it already has.
- The laptop asks for everything that changed since its last history id and applies it. History is ordered by (user, history_id), so this is another short range read at the head of one partition. Records say what changed (message added, labels added or removed, message deleted), not the full new state, so a client five changes behind reads five small records. The laptop stores the new history id only after applying everything.
- A tablet that was off for a month sends a history id that is no longer kept, and does a full sync. History is kept for a window (at least a week here) and then trimmed, so a very old id gets 404. The client then re-lists recent threads per label, reconciles its cache by message id, and starts again from the current history id. The public Gmail API documents the same contract: history ids are typically valid for at least a week, and a 404 means do a full sync.
- A botnet doubles inbound traffic. The scale-breaking case for email: a burst that is almost all junk. The design holds because most of it dies before DATA, and nothing slow runs until we have decided to own a message.
- Two hundred thousand compromised hosts start connecting, most from IPs we have never seen. Inbound jumps from 2.3 M to about 4.5 M messages a second. Known-bad IPs, from our own scores or public blocklists, get 554 and the connection closes. Unknown IPs get a neutral score, which means a budget of a few messages a minute and a cap on concurrent connections per IP and per /24.
- Unknown IPs that exceed their small budget are deferred with 421 and must retry later. A 4xx costs a real mail server nothing: it keeps the message and retries in minutes. Botnets usually do not retry, or retry so aggressively that the pattern itself lowers their score. Deferring is the safe answer under uncertainty because it refuses nothing permanently.
- Messages to made-up addresses are refused at RCPT TO, before any body is sent. Dictionary attacks guess addresses; each unknown recipient gets 550 5.1.1 for the price of one in-memory directory lookup. A sender with a high share of unknown recipients has its score cut at once, so its next connection is refused earlier.
- Messages forging a brand that publishes DMARC p=reject are refused after DATA. Botnets forge well-known brands in the From header. DMARC p=reject is the domain owner telling receivers to refuse mail in its name that fails authentication, so we answer 550. We never accepted it, so we generate no bounce; if the sender was a real server, it bounces to its own user.
- What survives reaches the queue, and the content classifier files most of it as Spam. Only accepted messages cost a replicated write and a model call. A campaign sending near-identical text from thousands of IPs is easy to cluster: one content fingerprint, thousands of sources. Accepted mail is labelled Spam, never bounced, because a bounce would go to a forged envelope sender.
- Spam-trap hits and campaign clusters lower the sending IPs’ scores within minutes, and the MX starts refusing them at connect. Spam traps are addresses that never signed up for anything, so any mail to them is unsolicited by definition, and they see a wave first. The feedback consumer writes new scores to the reputation store, which every MX reads on the next connection; user reports join the same loop. Closing the loop in minutes bounds the cost of a wave: its first minutes reach the classifier, the rest dies at connect.
- Send to a provider that is throttling us. The send path and its failure mode. Once the user sees "Sent" we owe them delivery or a bounce, while the receiving provider answers 421 and the retry clock runs for days.
- The user sends a message with an attachment to a friend at another provider and a colleague on our service. The attachment is stored as a blob, and the Sent copy plus one outbound job per recipient commit in one transaction on the sender’s shard, so we never show Sent for mail we will not try to deliver. Jobs carry a not-before time 10 seconds out: Undo send is cancelling a job that has not started.
- When the undo window passes, the colleague’s copy goes to the inbound queue and the friend’s to the outbound queue for their domain. Local delivery skips SMTP but not the filter, because compromised accounts send spam to our own users too. The external job lands in the queue for its destination alongside everything else we are sending there, so it inherits that destination’s current rate and backoff state.
- An outbound MTA takes the job at that destination’s current rate, signs it with DKIM and connects to the domain’s MX. The MTA resolves the domain’s MX records (cached), tries the lowest preference first and uses STARTTLS, required where the receiver publishes MTA-STS. The DKIM-Signature with d=ourmail.com lets the receiver verify our domain sent it unchanged; our SPF and DMARC records do the rest. The IP comes from the pool for normal users’ mail.
- The receiver answers 421: it is rate limiting us. The MTA reschedules the job and slows that destination down. A 4xx means try later: the next attempt moves out with exponential backoff and jitter (minutes at first, then every 30 to 60 minutes), and the destination’s rate is halved, then raised slowly while 250s come back. Every message for that destination slows down, which is the point: hammering a throttling provider gets us blocked. Other destinations’ queues are unaffected.
- A later attempt gets 250 and the job is done; had it still been failing after five days, or got a 5xx, the MTA would bounce it to the sender. Five days follows RFC 5321’s guidance that senders keep retrying for at least four to five days, and we also send a "still trying" notice after a few hours. A 5xx (550 no such user) ends retries at once. The bounce is a delivery status notification (RFC 3464) addressed to our own user, so it goes through the inbound queue like any local message.
- Search twenty years of mail. Search is per user by construction: a query never leaves one user’s index, which is why it can be fast over 20 years and safe without a user filter on every posting.
- The user opens the app, and the API warms that user’s search index while they read. Most users search rarely, so most indexes sit cold on cheap disk. Opening the app or focusing the search box loads the term dictionary and newest segments into memory, hiding a cold read of ~100 ms behind the user’s typing. An index that is never searched costs only disk.
- The user searches for invoices from Alice with an attachment, before 2019. The API parses operators into field terms: from:alice is a term in the from field, has:attachment and label:inbox are terms too, and before: is a range on the date. The query is routed to the one shard holding this user’s index, like every other call for this user.
- The index intersects the posting lists for those terms inside this user’s index. Documents are numbered in date order per user, so postings are date-sorted and the intersection walks from the newest end, stopping at 50 hits for the first page. The posting list for "invoice" in one mailbox has thousands of entries at most; in a global index it would have billions, filtered down to one user’s. Once segments are in memory this takes milliseconds.
- Messages newer than the index are scanned directly from the mailbox and merged in. The index records the last history id it applied. Messages after that, usually none to a handful, are read from the mailbox and matched in memory, so mail that arrived two seconds ago is findable. Search agrees with the inbox without putting indexing on the delivery path.
- The API turns the hits into thread summaries, drops anything deleted since, and returns the first page. Hits are message ids; the API reads their thread summaries and current labels from the same shard, drops messages deleted since the last merge, and groups by thread. A message moved to Trash a second ago therefore does not appear in a normal search, even though the index has not merged yet.
- In the background, the user’s delta segments are merged and deleted messages are purged from the index. Small segments from incremental indexing are merged per user, like an LSM tree, so a query reads a handful of segments, not hundreds. Deletes are tombstones until the merge, which is also when a user’s request to delete mail actually removes the terms. Merges are scheduled per shard during its quiet hours.
Deep dives
Owning a message: reject, defer or accept
A message looks suspicious. Do you refuse it in the SMTP session, accept it and file it as spam, or accept it and bounce it later?
SMTP gives the receiver one moment to say no: before it answers the end of DATA with 250. After that, RFC 5321 makes us responsible for delivering the message or reporting its failure, and the sender deletes its copy. At 2.3 M messages a second, three quarters of it junk, with a content model that takes tens of milliseconds and an attachment sandbox that takes seconds, we cannot run everything before that moment, and we cannot afford to store everything after it.
The answer depends on what each response does to the sender. A 5xx makes the sending server bounce the message to its own user: right when we are sure, and it costs us nothing. A 4xx makes it retry later: right when we are unsure or overloaded. A 250 followed by our own bounce sends mail to the envelope sender, which spam forges, so we would be mailing innocent people (backscatter). So the question becomes which checks are cheap and certain enough to run before 250, and what to do with everything else.
- Refuse in the session on cheap, certain signals; accept the rest and label uncertain mail Spam chosen
- Accept everything, decide later, bounce what is bad rejected
- Accept and silently drop what looks bad rejected
- Run the full content model and sandbox before replying situational: first contact from a brand-new sending domain, where we can answer 421 and score the message while the sender waits to retry
The answer: The 250 is our commit point, so it waits for the message to be appended to the replicated inbound queue (two of three replicas, in different zones), and an MX holds nothing on its own disk that matters. Before the 250 run only checks that take milliseconds and are near certain: IP and range reputation from an in-memory store, per-IP rate and connection limits, recipient existence, SPF, DKIM and DMARC (refusing where the domain publishes p=reject), size limits, banned attachment types and antivirus signature hits. Unknown or bursting senders get 421 rather than 5xx, because deferral is free for a real server and expensive for a botnet. Everything else runs after the 250 and produces a label, never a bounce: Inbox, Spam, or a quarantine that withholds a dangerous attachment. Our own bounces are reserved for real delivery failures, never for spam verdicts. Google’s public guidance for bulk senders has the same shape: unauthenticated mail and excessive rates are refused or deferred in the SMTP session, while content judgements land in the Spam folder.
The MX crashes after the queue append but before the sender receives 250. What happens?
The sender never saw 250, so it keeps the message and retries, and we receive it twice. This is SMTP’s classic duplicate, and it is the right way to fail: a duplicate is recoverable, a loss is not. Delivery dedupes on (recipient, Message-ID, body hash) for a few days, so the user sees one copy. A message without a Message-ID gets one at the edge and can only be deduped by body hash.
The inbound queue is slow or down. What does the edge do?
Stop answering 250 and answer 421 instead, so senders keep the mail and retry. That turns our outage into a delay, which SMTP is built for: senders retry for days. What we must never do is fall back to accepting onto an MX’s local disk without replication, because then an MX failure loses mail we promised to deliver. Queue capacity per region is alarmed long before this point.
Forwarding and mailing lists break SPF and sometimes DKIM. Doesn’t strict DMARC refuse legitimate mail?
Yes, that is the known weak spot. A forwarder sends from its own IP, so SPF fails for the original domain, and a list that adds a footer or tags the subject breaks DKIM. Mitigations: honour ARC (RFC 8617), where a trusted intermediary records the authentication results it saw; treat DMARC failures from known forwarders and large lists as a spam signal rather than a refusal; and recognise users’ own forwarding setups. Lists that rewrite From to their own domain avoid the problem.
Why not greylist everything, deferring the first message from any new sender?
It beats simple bots, but it delays every first email from a new correspondent by the sender’s retry interval, often 15 to 30 minutes, which users hate for sign-up codes and password resets. Serious spam infrastructure retries correctly, so the benefit has shrunk. I would defer only unknown IPs that exceed a small budget, not every first contact.
How do you know the edge is refusing the right things?
Sample refused sessions and replay them through the full classifier offline: the share it calls legitimate is the edge’s false-reject rate, and it must be tiny. Track refuse and defer rates by reason, by sending provider and by region, and alert on sudden changes: a spike in DMARC refusals for one big sender usually means they broke their DKIM, not that they turned to spam. Postmaster feedback to large senders closes the loop.
The mailbox store and its shard key
How do you store 90 trillion messages so the inbox is one range read and a label change is a tiny write?
A mailbox is used in a few narrow ways: list threads with a label, newest first; open one thread; change labels and read state on many messages at once; append new mail; ask what changed since N. Every one of them concerns exactly one user. The bytes are lopsided: bodies and attachments are about 95 % of 4.5 EB and rarely read after the first week, while metadata is a few percent of the bytes and nearly all of the operations.
Two observations shape the design. First, no mailbox needs more than one machine: the heaviest 20-year mailbox has about a million messages and a gigabyte of metadata, so partitioning by user never produces a key too big for its shard, unlike a celebrity in a social graph. Second, messages never change after delivery but their labels change constantly, so the two should not be stored together.
- Shard by user id: immutable message records, small mutable metadata, bodies in blob storage chosen
- Mailbox files (Maildir or mbox) on file servers situational: a provider with thousands to a few million mailboxes, where Dovecot-style storage on replicated file servers is simple and proven
- One global message table partitioned by message id rejected
- Labels and flags stored inside the message record rejected
The answer: A sharded store whose shard key is user_id, with every row for a user on one shard: sharded MySQL with Vitess, or a Spanner-class database with the user as the root of interleaved tables. What matters is single-shard transactions, not the brand. Tables: messages (user, msg_id) with parsed headers, size, snippet and blob references, written once; meta (user, msg_id) with labels, read and starred flags and thread_id; threads (user, thread_id) with the summary the list shows; label_index (user, label, last_ts desc, thread_id); counters (user, label); history (user, history_id). A delivery or a label change updates its rows and appends one history record in one transaction. Thread ids are assigned at delivery from In-Reply-To and References against the user’s Message-ID index, falling back to the normalised subject with an overlapping participant in a recent window. Each shard is a replicated group of tens of thousands of users across zones; a growing shard is split by moving users. Bodies and attachments go to the blob store with the other 95 % of the bytes.
How exactly do you pick the thread for a new message?
Look up its In-Reply-To and References ids in this user’s (user, Message-ID) → thread index; the first hit wins. With no hit, normalise the subject (strip Re:, Fwd:, AW: and list tags like [team]) and look for a thread with the same normalised subject and an overlapping participant in the last few weeks; subject alone would merge every "Invoice" ever received. When a reply arrives before its parent, it starts a thread and records each id it references, so the parent finds that thread when it lands. A changed subject starts a new thread, and very long threads are capped (say at 100 messages) so a thread stays bounded.
A shard holding 30,000 users loses its leader. What do those users see?
Shards are replicated groups with consensus across three zones, so a leader election takes a few seconds, during which those users’ writes wait and reads can be served slightly stale from a replica. Their inbound mail waits in the inbound queue, which is exactly why the queue exists. Because no user spans shards, a failure affects a known set of users completely and everyone else not at all.
Why not Cassandra with the user as the partition key?
It fits the access pattern: partition by user, cluster by label and time. The gap is transactions: a delivery writes the message, label index, counter and history record together, and history ids must increase without races. Cassandra gives that only through lightweight transactions on every write, which are slow, or through idempotent rows written in a careful order with the history record last. It can be made to work; a store with single-shard transactions makes it much easier to be correct.
How do unread counts stay right across 1.5 B mailboxes?
They are counters updated in the same transaction as each label or read-state change, so they are exact within a shard by construction. Bugs still happen, so a background job recomputes counts from label_index for a sample of users, and for any user who reports a wrong badge, and repairs drift. The badge reads the counter, never a count query.
One account receives 2 million messages a day, say a big company’s catch-all address. Does partitioning by user still work?
It strains it: 2 M a day is about 23 writes a second sustained, which one shard handles, but the mailbox grows by tens of gigabytes a month and blows through any quota. This is a product limit as much as a storage one: per-account daily receive limits, a quota that rejects with 452 or 552 once full, and steering such workloads to a business product with retention policies. The design should not be bent for the one case the quota exists to prevent.
Attachments: dedupe without leaking
The same 2 MB PDF arrives for a million users. Do you store it once, and what does sharing one copy across users cost you in privacy?
Attachments are most of the bytes in a mailbox, and many are identical: a newsletter’s PDF sent separately to every subscriber, the file that comes back attached to every reply in a thread, the same invoice template every month. Storing a 2 MB file once for a million recipients instead of a million times saves 2 TB from one campaign, and content hashing makes identical bytes easy to find.
Sharing one stored copy creates three obligations. Access must never follow from knowing a hash, only from the user’s own message. Deleting must remove one user’s reference without touching anyone else’s copy, and the last reference must really free the bytes. And a per-user encryption key cannot protect a blob that many users share. The well-known attack on dedupe, probing whether a file exists by uploading only its hash, does not apply as long as every byte still arrives over SMTP or a full upload: the dedupe must stay on the server.
- Server-side dedupe across all users by keyed content hash, with reference sets chosen
- Per-user dedupe only situational: a tenant that holds its own encryption keys or forbids sharing stored bytes with other accounts
- Client-side dedupe: upload only if the server lacks this hash rejected
- No dedupe rejected
The answer: Split the MIME message at delivery: the text and HTML bodies go into one compressed object per message, and each attachment above ~64 KB becomes a blob whose id is HMAC-SHA256 of its bytes under a service secret, so a blob id that leaks into a log cannot be matched against a known file outside our systems. Each blob keeps a reference set of (user, msg_id) rows; delivery adds its reference before the mailbox transaction commits, and deletion removes it after the message is purged, so any failure leaves an extra reference (a leak the GC finds later) and never a missing one. A blob whose set is empty is deleted only after a grace period of days and a re-check. Every blob is encrypted with its own random data key, wrapped by a key-management service. Clients never see blob ids: an attachment is fetched as (message, part) and authorised through the user’s own message record. Quota charges each user the full size, and tenants that hold their own keys get their own dedupe domain.
Why a reference set instead of a counter?
Delivery is at least once. A retry that increments a counter twice leaves a count that never reaches zero, a lost decrement leaves it stuck, and you cannot tell which references are real. With a set of (user, msg_id), adding the same reference twice is a no-op, and the GC can verify every entry against the mailbox it names. For a blob with millions of references the set is stored as its own rows under the blob id, so it is a range, not one giant value.
A user deletes their account. Is their attachment gone if 10,000 other people received the same file?
Their copy is: their messages and references are removed, and nothing is reachable from their account. The bytes remain only because 10,000 other people independently received the same file, and those copies are theirs, exactly as if each had been stored separately. When the last reference goes, the blob is deleted after the grace period.
Could someone learn that another user has a file by sending it to themselves and timing delivery?
Not in this design. The sender always transmits every byte, delivery does the same work whether the blob is new or already stored, and no response reveals which it was. If the store wrote measurably faster on a hit, we would equalise that path. The attack that hurt file-sync products needed the client to skip the upload, which email never lets it do.
A 25 MB attachment is sent in one message to 500 of our users. What happens?
The MX receives it once, the queue stores it once, and delivery writes one blob and adds 500 references, one in each recipient’s transaction: 500 small mailbox writes and one 25 MB blob write instead of 12.5 GB. Each user is still charged 25 MB of quota, because they think of it as their own copy and our saving should not change what they see.
Where does malware scanning fit with dedupe?
Scan results attach to the blob, so a file sent to a million users is detonated once. If a new signature later flags it, every message referencing it can have the attachment withheld at read time without touching a million mailboxes. Sharing one copy is a real security benefit here.
Search: one index per user
Why not put all mail in one big search cluster, the way web search does?
There are about 90 trillion messages and a full-text index over them of ~150 PB. A query must answer in under a second over up to 20 years and a million messages for a heavy user, and every query is about exactly one mailbox: nobody searches anyone else’s mail. Most users search rarely; a few search all day.
Web search partitions by document and sends every query to every shard, because any document might match. Email is the opposite case: the user id is part of every query, so the natural partition is the user. The posting list for "invoice" across all users has billions of entries, of which one user owns a few hundred; per user it has only those few hundred to begin with.
- A per-user inverted index stored with the mailbox and updated from its history chosen
- A shared search cluster routed by user id (Elasticsearch with routing) situational: millions rather than billions of mailboxes, where an off-the-shelf engine is cheaper to run than a custom one
- One global index over all mail rejected
- No index: scan the mailbox at query time rejected
The answer: Each user has an inverted index on the same shard as their mailbox, with fields for from, to, cc, subject, body text, attachment names and labels, plus terms for operators such as has:attachment and is:unread. Documents are numbered in date order per user, so posting lists are date-sorted and the default newest-first query stops at the first page. An indexer co-located with the shard tails each user’s history: new messages go into a small delta segment, label changes update the label terms, and a background merge per user keeps the index to a few segments and purges deletes. A query reads the index plus a direct scan of messages newer than the index’s last applied history id, so mail is searchable the moment it is in the inbox. Indexes of users who have not searched for weeks live on cheaper disk and are prefetched when the app opens. The scan-the-mailbox alternative survives only in that small role: covering the few messages the indexer has not reached.
A user with a million-message mailbox searches for a word that matches most of it. What happens?
The query only needs the newest 50 hits, and postings are in date order, so the engine walks from the newest end and stops after 50: cost is bounded by the page size, not by the match count. A result count like "about 120,000" is an estimate from the posting list length, not computed exactly.
Labels change constantly. Do label changes rewrite the index?
No. Text never changes after delivery, so the large structures are append-only per user; labels live in a small separate structure per user, updated in place from history, and are re-checked against current metadata when hits are hydrated. A search for label:inbox therefore reflects an archive within a second.
A bug corrupts some users’ indexes. How do you recover?
The index is derived from the mailbox, so a job rebuilds an affected user’s index from their messages, swaps it in, and replays any history since the rebuild started. Because indexes are per user, the repair is small and local. A fleet-wide format change works the same way, shard by shard in the background, serving the old format until each user’s new index is ready.
Do you index the contents of attachments?
Asynchronously and selectively. Extracting text from PDFs and documents is expensive, attachments are shared across users, and most are never searched. Extract once per blob (dedupe helps again), store the text with the blob, and add it to each referencing user’s index lazily; attachment names are indexed at delivery because that is what people mostly search for.
How would you measure search quality, not just its speed?
Log which result the user opened and at what rank, and how often a search is refined or abandoned. The success rate is the share of searches where the user opens a result from the first page; a drop after a change to parsing or ranking is the alarm. Because the data is private, offline evaluation uses consenting test accounts and synthetic mailboxes, not real users’ mail.
Keeping every device in step
A phone, a laptop and an IMAP client show the same mailbox. How does each one find out what changed, and what happens when a notification is lost?
A user has two or three devices, each caching their recent mail. Something changes about once a minute: new mail, an archive on another device, a filter applying a label. Re-listing the inbox on every change would turn ~700 k sync calls a second into 700 k full listings, and polling adds either latency or load. Labels make it subtler: an archive changes a message’s label set, which an IMAP client sees as the message leaving a folder.
The key is a per-mailbox, monotonically increasing change id written in the same transaction as each change, which the public Gmail API exposes as historyId. A client that has applied everything up to N needs only the changes after N, in order. Notifications then only have to say "something changed, the latest is M", and can be lost, duplicated or coalesced without harm.
- Per-mailbox history id with a change log, plus content-free push hints chosen
- Changed-since timestamps rejected
- Full diff: the client sends what it has, the server compares situational: initial sync, and the fallback after a client’s history id has expired
- Push the changes themselves to devices rejected
The answer: Every change to a mailbox (message added, labels added or removed, message deleted) appends a history record with that user’s next history id, in the same transaction as the change. The API serves "changes since N" as a range read; records are kept for at least a week, and a client with an older id gets 404 and does a full sync (re-list recent threads per label, reconcile by message id, adopt the current id), which is the same contract the public Gmail API documents. Hints carry only { user, history_id }: APNs or FCM for phones, a long-lived channel for web tabs, IDLE for IMAP clients. Client changes are label set operations (add Work, remove INBOX), so retrying them is harmless, and when two devices touch the same label the server’s history order decides. IMAP is a translation layer: each label is a folder with its own UIDs, history records become EXISTS and EXPUNGE responses, and CONDSTORE mod-sequences map onto history ids.
The phone archives a thread offline while the laptop labels it Work. Who wins?
Both, because they touch different labels: the phone removes INBOX, the laptop adds Work, and set operations on different elements commute. When both touch the same label (one archives, the other moves it back to the inbox), the server applies them in arrival order and both devices converge on its state through history. Last writer wins is acceptable for labels; it would not be for new mail, and new mail cannot be lost this way because only the server adds messages.
Why must the history id be assigned inside the same transaction as the change?
Otherwise a client can read history up to N, and then a change commits with an id below N that it will never ask for. Assigning the id inside the transaction on the user’s shard makes ids appear in commit order, so "everything after N" is complete. Ids may have gaps (the public Gmail API says its ids increase but are not contiguous), which is fine: completeness needs order, not density.
A busy user has 40 changes a minute and five devices. Isn’t that a lot of pushes?
The notifier coalesces per mailbox, at most one hint every few seconds, carrying the latest id. Foreground devices hold a live channel and fetch at once; a backgrounded phone gets a push only for mail that should notify the user, and syncs the rest when opened. The work per change stays a read of a few rows at the head of one user’s history.
IMAP clients expect folders with stable UIDs. How do labels map without breaking them?
Each label is exposed as a folder, and the gateway keeps a UID sequence per folder: gaining a label gives the message a new UID in that folder, losing it is an EXPUNGE there. Moving a message between folders becomes adding one label and removing another, and deleting from a label folder removes that label rather than the message. Gmail’s own IMAP service documents a similar mapping, with an All Mail folder holding everything.
How would you know sync is broken for some users?
Canary accounts on every client version make changes and measure how long other devices take to reflect them. Track the rate of 404 full syncs, since a spike means history is trimmed too aggressively or clients are stuck, and have a sample of clients report a checksum of their cached labels for recent threads to compare with the server’s.
Sending without getting blocked
Yahoo starts answering 421 to a third of your mail. How do you keep delivering to them, and to everyone else, without getting blocklisted?
Users send about 3 B messages a day (~35 k/s), and roughly half go to other providers. Each receiver throttles by sending IP and domain, and its patience depends on our reputation: complaint rates, mail to addresses that do not exist, sudden volume. SMTP gives two kinds of refusal: 4xx means try later, 5xx means never. A sender that retries hard after 4xx, or keeps sending to dead addresses, gets blocked, and then every user’s mail to that provider fails.
The pressure is asymmetric. One throttling destination must not slow mail to everyone else, and a few compromised accounts sending spam must not ruin delivery for 1.5 B honest users. So the structure is isolation, by destination and by class of sender, and the behaviour is politeness per destination.
- Per-destination queues with adaptive rates, backoff on 4xx, bounce on 5xx, separate IP pools chosen
- One shared FIFO queue with retries rejected
- Relay through a commercial email provider situational: a new or small provider with no sending reputation of its own yet
- Send synchronously from the API while the user waits rejected
The answer: After the Sent copy commits, each external recipient becomes a job in the queue for its destination, grouped by the MX host the domain resolves to, so all domains hosted by one provider share one politeness budget. Outbound MTAs take jobs at each destination’s current rate, which starts from history, halves on a burst of 4xx and grows slowly while 250s come back; tokens live in a shared limiter and MTAs lease them in small batches. A 4xx reschedules the job with exponential backoff and jitter for up to five days, with a "still trying" notice to the sender after a few hours; a 5xx, or the five-day limit, produces a bounce to our own user. Every message is DKIM-signed with d=ourmail.com and a rotating selector, and our SPF and DMARC records cover the sending IPs. Outbound mail is scored like inbound: accounts that look compromised are rate limited and sent from a separate IP pool, and new IPs are warmed up over weeks. Receivers’ complaint feedback loops and postmaster dashboards are a primary signal, not an afterthought.
An account is compromised and sends 10,000 spam messages in an hour. What stops it?
Per-account limits first: a daily recipient cap and a rate limit that is tighter for new or unusual senders. The outbound filter scores content with the same models as inbound, so a burst of near-identical messages to strangers scores high, is held or sent from the risky pool, and the account is challenged to re-authenticate or suspended. The main pool’s reputation never sees most of it.
The remote server answers 250, then later sends a bounce anyway. What do you do?
That is an asynchronous bounce: the receiver accepted, then failed internally. It arrives as an inbound delivery status notification to our envelope sender, so we use a VERP-style envelope sender (bounce+<message id>@ourmail.com) to match it to the original message and recipient. It is delivered to the user like any bounce, and repeated hard bounces for an address feed a warning the next time they write to it.
How do 1,000 MTAs share one rate limit for Yahoo?
A shared limiter holds a token bucket per destination, replicated in memory, and each MTA leases a small batch of tokens at a time, so no MTA makes a round trip per message. A 4xx seen by any MTA lowers the shared rate. If the limiter is unavailable, MTAs fall back to a small fixed local share rather than sending unthrottled.
How do you start sending from a brand-new IP range?
Warm it up: a small daily volume to each large provider at first, choosing the mail most likely to be wanted (replies to existing threads, mail to frequent contacts), doubling over days to weeks while bounce and complaint rates stay low. Receivers judge an IP by its history, and a new IP that suddenly sends a flood looks exactly like a spammer.
What should "delivered" mean in the UI?
For external mail, nothing more than sent: an SMTP 250 means the receiving server took responsibility, not that the recipient read it or that it reached their inbox rather than their spam folder. We show the message as sent and report failures, the bounces, explicitly. Claiming more would be inventing information we do not have.
The theory behind it
- Designing an email inbox (Gmail-style): The backend of a webmail service: receiving mail over SMTP, spam filtering, storing messages and attachments, threading into conversations, labels and folders, per-user search, sync with IMAP and mobile clients, quotas, deduplication and retention.
- Sending email at scale: How transactional and bulk email systems work: SMTP and email providers, deliverability with SPF, DKIM and DMARC, IP and domain reputation, warm-up, bounces and complaints, suppression lists, unsubscribe handling, queues and rate limits per provider, templates, tracking and privacy.
- Search, indexing and autocomplete: Inverted indexes, analysis and ranking, prefix search with tries, filtering by permissions, index freshness, and when vector search actually belongs in the answer.