Design Pastebin

Store and serve 10 M text pastes a day behind unguessable links: expiry that is never late, burn after reading, and abuse takedown.

Last updated 2026-10-27. Difficulty: medium. Patterns: object-storage, expiry, caching, privacy, abuse. Reported at Amazon, Microsoft, Google, Meta, Dropbox, Atlassian.

Functional requirements

• Create a paste. Plain text up to 10 MB (typically ~10 KB), with an optional title and language, anonymous or signed in. Returns a page link, a raw link and, for anonymous pastes, a delete token shown once.
• Read it rendered or raw. A page with syntax highlighting and line numbers, a raw text view for curl and scripts, and a download. The text is never interpreted as HTML anywhere.
• Visibility: public, unlisted, private. Public pastes appear in the public feed and in search engines once scanned; unlisted ones are readable by anyone with the link and listed nowhere; private ones only by the signed-in owner, and they are a 404 for everyone else.
• Expiry. 10 minutes, 1 hour, 1 day, 1 week, 1 month, 6 months, 1 year or never. An expired paste is never served again, from any cache, and its bytes are reclaimed.
• Burn after reading. Shown to exactly one reader, then gone. A chat app’s link preview or an email scanner must not consume it, and two people opening it at once must not both see it.
• Delete and takedown. Owners delete with their session or the delete token; trust and safety takes down abusive pastes. Both take effect at every CDN edge within minutes, and a taken-down paste cannot simply be pasted again.
• View counts. Shown on the page, approximate and a few minutes behind, even though most views are served by the CDN and never reach us.
• Abuse controls. Rate limits per IP and per account, asynchronous scanning for leaked secrets, malware and spam, a report button, and takedown backed by a hash blocklist.
• Out of scope. Editing (pastes are immutable; an edit creates a new paste), comments and forks, full-text search across all pastes, end-to-end encrypted pastes (discussed under dedupe), and the moderation console itself.

Non-functional requirements

• Read latency (p50 < 50 ms · p99 < 300 ms). Edge hits answer in tens of milliseconds anywhere. The p99 comes from origin misses on blob-backed pastes, which add an object storage read; storing small pastes inline keeps most misses to one key-value lookup.
• Availability (99.95 % reads · 99.9 % writes). Paste links sit in bug reports, chat threads and CI logs, and a read outage breaks all of them at once. A failed create costs one person a retry.
• Durability until expiry (no paste lost before it expires). Rows replicated three ways across zones, blobs in object storage designed for eleven nines. Deleting late is a cost; deleting early is data loss.
• Expiry is exact (0 reads after expires_at). People put secrets in 10-minute pastes. Correctness comes from a check on every read and cache lifetimes capped at the remaining life; reclaiming the storage may lag by hours without anyone being able to tell.
• Unlisted and private stay private (P(guess) < 10⁻⁶ · never in a shared cache). This is the requirement that drives the hardest trade-off. The CDN wants to hold everything for a month, while unlisted links must be unguessable, private and burn content must never sit in a shared cache, and deletes and takedowns must reach every edge. Most of the caching rules exist to reconcile the two.
• Burn after reading (at most one reader). One successful reveal under concurrent requests, link previewers and retries. Losing the content when the winner’s connection drops is acceptable; showing it twice is not.
• Abuse response (scan < 1 min · takedown < 5 min globally). Paste sites are used to leak credentials and stage malware. Public pastes reach the public feed only after the scan, and a takedown must clear every CDN edge, not just the database.
• Scale (10 M writes/day · 150 M reads/day · 5 years). Average 10 KB, 10 MB maximum. Half of all pastes never expire and are kept for at least five years; 65 % of those are unlisted.

Back-of-envelope estimates

• Writes per second: ~120 avg · ~350 peak. 10 M pastes/day ÷ 86 400 ≈ 116/s, and about 3× that at the daily peak: ~350/s. At ~10 KB each that is ~1.2 MB/s of incoming text. Writes are not the hard part.
• Reads per second: ~1.7 k avg · ~5 k peak. 15 reads per paste: 150 M reads/day ÷ 86 400 ≈ 1.7 k/s, ~5 k/s at peak. The head matters more than the average: one paste linked from a front page can draw 20 k/s on its own.
• Bytes per paste, stored: 10 KB raw → ~3 KB. Median ~2 KB, mean 10 KB: 90 % of pastes average 2.5 KB and the other 10 % (logs, dumps) average ~78 KB. zstd shrinks short snippets ~2× and logs ~4×: 0.9 × 1.25 KB + 0.1 × 19.4 KB ≈ 3.1 KB, about 3.3× overall.
• Storage after 5 years: ~28 TB compressed. 10 M/day × 10 KB = 100 GB/day, ~180 TB raw over five years. Half of all pastes expire within a year, so ~9.1 B never-expiring pastes remain: 9.1 B × 3.1 KB ≈ 28 TB, of which ~10 TB sits inline in the store and ~18 TB in object storage (~14 TB after dedupe). Storage is not the hard part.
• Blob writes per day: ~1 M (vs 10 M). Keeping content of 8 KB or less (compressed) inline keeps ~90 % of pastes out of object storage: 10 M × 10 % = 1 M PUTs/day instead of 10 M. At ~$5 per million PUTs that is $5 a day rather than $50, and nine in ten origin reads become a single key-value lookup.
• Id length for unlisted pastes: 16 base62 chars (~95 bits). 65 % of 9.1 B retained pastes are unlisted: N ≈ 6 B. An attacker with G = 10^12 guesses (30 k/s for a year from a botnet) and a target P(any hit) ≤ 10^-6 needs a space ≥ G × N ÷ P = 6 × 10^27. 62^15 ≈ 7.7 × 10^26 is too small; 62^16 ≈ 4.8 × 10^28 gives P ≈ 1.3 × 10^-7. A shortener’s 7 characters would give one hit per ~590 guesses.
• Origin reads after the CDN: ~380/s avg. 92 % of reads are public or unlisted, so cacheable. With an 85 % edge hit rate on those, origin sees 0.92 × 0.15 + 0.08 (private and burn, never cached) = 22 % of 1.7 k/s ≈ 380/s, ~1.1 k/s at peak. One key-value read each, so a modest store handles it.
• Expirations to sweep: ~5 M rows/day · ~0.5 M blobs. At steady state everything given an expiry expires: 10 M × 50 % = 5 M rows/day, ~60/s. One in ten has a blob: 0.5 M/day, or ~500 batched delete calls of 1,000 keys. Small, as long as the sweeper reads an index ordered by expiry time instead of scanning 9 B rows.

Components

• Browser / CLI (web · curl · editor plugins): Creates pastes from the web form, curl or an editor plugin, and reads them as a page or as raw text. Highlights code itself in a Web Worker from the stored language tag, so the server never turns user text into HTML. Reveals a burn paste only on an explicit click, which sends a POST.
• Raw content host (cookieless domain · text/plain): Origin for pasteusercontent.example, a separate registrable domain that never receives a cookie. Resolves the paste through the paste service, so expiry, visibility and takedown are checked in one place, then returns text/plain with nosniff and a sandboxing CSP, so nothing in a paste can run. A private raw view needs a short-lived signed link issued by the app.
• CDN edge (two hosts · surrogate keys): Serves paste.example (pages and API) and pasteusercontent.example (raw text). Caches public and unlisted responses for 30 days or the paste’s remaining life, whichever is shorter, keyed by id and tagged p:{id} for purges, and coalesces concurrent misses. Passes POSTs and the private and burn paths straight to origin, and streams its access logs to the view counter.
• View counter (CDN logs · 1-min batches): Counts views from the CDN’s log stream, because ~85 % of views of public and unlisted pastes never reach the origin. Sums per paste per minute and applies one increment per paste per minute, so a viral paste costs one write a minute instead of 20 k a second. Approximate by design: known bots are filtered out and the number lags by minutes.
• Object storage (S3-class · lifecycle rules): Content above 8 KB compressed: ~10 % of pastes but ~63 % of stored bytes. Never-expiring pastes live under cas/{scope}/{sha256}, shared within a scope and reference counted; expiring ones under exp/{class}/{id}, unshared, with lifecycle rules that delete each class a day after its maximum life. Private to the paste service, so a blob key is never a URL.
• Paste service (stateless · zstd · checks): Owns create, read, reveal, delete and report. On write it rate limits, validates UTF-8 and size, hashes, checks the blocklist, compresses, chooses inline or blob, and commits the row last. On read it is the one place the expiry check, visibility rules and takedown state are enforced, and it sets cache lifetimes capped at the paste’s remaining life.
• Rate limiter (Redis token buckets): Token buckets per IP (grouped by /24 for IPv4 and /64 for IPv6, since one connection usually owns at least a whole /64), per account and per tier, weighted by size so a 10 MB paste costs more than a 1 KB one. Also counts 404s on well-formed ids per network, which is how an id scan shows up. Fails open with conservative local limits if Redis is down.
• Expiry sweeper (expiry index · batch deletes): Reads closed hours of the expiry index across 64 shards, deletes expired rows with conditional deletes and their blobs in batches of 1,000 keys. Also consumes delete, burn and takedown events: purges the CDN by surrogate key, decrements reference counts on shared blobs, and deletes blobs whose count has been zero for 24 hours. Every operation is idempotent, so it can rerun or overlap the lifecycle rules safely.
• Paste store (DynamoDB / Cassandra · by id): pastes keyed by id: owner, visibility, expires_at, burn flag, state, language, SHA-256, sizes, and either the zstd content itself (8 KB or less) or a blob key. Beside it: an owner index, the expiry index by (hour, shard), blob reference counts, the hash blocklist, reports and view counters. Point lookups and conditional writes are all it must do well; ~13 TB after five years, ~40 TB with three replicas.
• Change stream (CDC → Kafka · key = id): Every committed change to a paste row (created, reported, deleted, burned, taken down), published from the store’s change log, so an event exists if and only if the write committed. Trust and safety and the sweeper consume it independently. Keyed by paste id so one paste’s events stay in order.
• Trust & safety (scanners · reports · takedown): Scans every new paste within a minute: secret detectors (cloud keys, tokens, private key blocks), credential-dump and spam heuristics, and decoded payloads checked against malware feeds. Public pastes enter the public feed only after the scan passes. Reports and scanner hits are queued by severity times current view rate; a takedown flips the row to taken_down and adds its hash to the blocklist.
• Threat intel (URL and malware reputation): Third-party reputation services: Safe Browsing-style lists for phishing links inside pastes and malware hash feeds for payloads decoded from base64 or hex. Called asynchronously with cached verdicts, never on the write path, because their verdicts arrive after the fact and paste sites are a favourite staging area for malware.

User flows

1. Create a paste. The write path. Easy at 120 a second; the design is in what happens before the row is written: limits, validation, the blocklist, compression, the inline-or-blob decision and an id nobody can guess.
   1. Client posts the text with its options. The CDN passes POSTs through uncached. Unlike Dropbox there is no presigned upload straight to storage: 10 MB fits in one request, and the service has to validate, hash and compress the bytes anyway.
   2. Paste service checks the rate limits for this IP and this account. Anonymous creators get about 10 pastes a minute and 100 a day per network, counted per /24 for IPv4 and per /64 for IPv6, because one home connection is usually given at least a whole /64. Accounts get limits by tier. The cost is weighted by size, so ten 10 MB dumps cost far more than ten one-line snippets.
   3. Service validates the text, hashes it and checks the hash against the takedown blocklist. Invalid UTF-8 is rejected: this is a text service, and refusing binary blocks raw executables outright. The SHA-256 of the raw bytes is checked against the hashes of every paste ever taken down, so an exact re-paste is refused before anything is stored. Near-duplicates are the scanner’s job, later.
   4. Service compresses with zstd; 8 KB or less stays inline, anything larger is written to object storage first. zstd at a fast level costs microseconds for a typical paste and shrinks text 2 to 4×. The threshold applies to the compressed size, so ~90 % of pastes never touch object storage. The blob is written before the row (commit last): a crash leaves an unreachable orphan for the lifecycle rules, never a row pointing at nothing.
   5. Service draws a random id and writes the row, the owner index and the expiry index in one transaction. The id is 16 base62 characters from a CSPRNG, about 95 bits, so the write needs no coordination: put-if-absent catches the roughly one-in-a-billion collision. The expiry index entry goes in the same transaction, so no paste can exist without a way to expire.
   6. Service returns the links; an anonymous creator gets a delete token, once. Only a hash of the delete token is stored, so a leak of the store does not let anyone delete pastes. The raw link is on a different registrable domain from the page link.
   7. The change stream publishes the new paste and trust and safety scans it in the background. Scanning is asynchronous so a slow detector never delays a create. The link works immediately, but a public paste joins the public feed only after the scan passes, about a minute later, because scrapers watch the feed for leaked credentials.
2. Read a paste, rendered and raw. Most reads end at the CDN. The origin path is one key-value lookup for nine pastes in ten, and it is the only place expiry, visibility and takedown are checked.
   1. Browser requests the paste page and the CDN serves it from cache. About 85 % of reads of public and unlisted pastes end here. The cached HTML is identical for every viewer: the signed-in header and the owner’s delete button load from a separate, uncached API call, and the CDN ignores cookies in the cache key.
   2. On a miss, the CDN asks the paste service, which reads the row by id. Concurrent misses for the same id are coalesced at the edge, so a burst becomes one origin request per PoP. For 90 % of pastes the content is in the row, so this is the only storage read.
   3. Service applies the read checks: live state, not expired, and visible to this caller. expires_at is compared with the clock on every read, so an expired row that the sweeper has not reached yet still returns 410. A taken-down paste returns 451. A private paste returns 404, not 403, to anyone but its owner, so a guess cannot even confirm that it exists.
   4. For a paste above 8 KB, the service streams the content from object storage. Streamed, not buffered, so a 10 MB paste does not sit in service memory. This hop adds tens of milliseconds to first byte and is where the p99 comes from, which is why it is reserved for the 10 % of pastes large enough to justify it.
   5. Service returns the page with cache lifetimes capped at the paste’s remaining life. The CDN may keep it for up to 30 days, browsers and other proxies for 5 minutes, and neither past expires_at, so expiry never needs a purge. The surrogate key lets one purge clear the page, raw and download variants together. The text is embedded escaped, and the page’s CSP forbids inline script.
   6. A script fetches the raw link; the raw host resolves it through the paste service and returns plain text. The raw host lives on a separate registrable domain, not a subdomain, so no cookie for paste.example is ever sent to it and it is not same-site with the app. If a browser ever rendered a raw paste as HTML, the script would run with no session to steal. nosniff also stops another site from loading a raw paste as a script or stylesheet.
   7. The CDN’s access logs feed the view counter, which adds each paste’s views to its counter once a minute. The origin cannot count views it never sees. One increment per paste per minute caps the write rate of a viral paste at one a minute. The number on the page comes from a separate call cached for 60 seconds, so the immutable page never carries a stale count.
3. A leaked dump goes viral, then is taken down. The scale-breaking case and the abuse case at once: 20 k requests a second for one paste, which then has to vanish from every edge. The CDN absorbs the load; the hard part is making a takedown stick.
   1. A public paste full of email and password pairs is linked from a large forum and draws 20 k requests a second. Spread across a few hundred PoPs, that is under 100 requests a second at each, trivial for the edge. The paste was created anonymously ten minutes ago with expiry set to never.
   2. Each PoP misses once; coalescing and the CDN’s upper tier turn that into a handful of origin reads. With a shield tier per region, origin sees about one fetch per region, then nothing for up to 30 days. Without coalescing, the first second would send every concurrent miss to origin at once.
   3. Readers report it; reports are written to the store and reach trust and safety on the change stream. The scanner had already flagged it as a likely credential dump (thousands of lines shaped like email:password) and kept it out of the public feed, but the link spread anyway. The queue orders reports by severity times current view rate, so a paste read 20 k times a second jumps to the front.
   4. Trust and safety checks reputations and takes the paste down: the row flips to taken_down and its hash joins the blocklist. For a clear credential dump with scanner and report signals agreeing, an automatic rule acts and a moderator reviews afterwards; ambiguous cases wait for a person. The state flip is conditional, so a takedown never resurrects a paste the owner already deleted.
   5. The takedown reaches the sweeper on the change stream, which purges every cached variant by surrogate key. Page, raw and download responses all carry p:{id}, so one purge clears them everywhere in seconds. Copies in browsers and other proxies live at most 5 minutes, which is exactly why their lifetime is short and only our CDN gets the long one.
   6. The next requests miss and get a 451, cached briefly at the edge; an exact re-paste is refused by the blocklist. Caching the 451 for 60 seconds matters: right after the purge, 20 k requests a second all miss at once, and without a cacheable answer they would all reach origin. Re-pastes with a line changed get past the exact hash and are caught by the scanner’s similarity check instead.
   7. The content is kept under a legal hold for the period policy requires, then the sweeper deletes it. A taken-down paste is unreadable at once but not necessarily erased at once: some takedowns must be preserved for a fixed time for law enforcement. The hold is an attribute on the row, and the sweeper deletes the bytes when it lapses.
4. Burn after reading: two readers and a bot. The correctness case. A link preview bot, two people clicking at once and a CDN that would happily cache or share one response, against a promise of at most one reader.
   1. The link to a burn paste is shared in a chat app, whose preview bot fetches it. A GET on a burn paste never returns its content, only an interstitial saying it will be deleted after viewing. Preview bots, email security scanners and browser prefetchers only ever see that page, so they consume nothing.
   2. Two people click Show at the same moment; both POSTs pass through the CDN to the paste service. POSTs are never cached or coalesced by the CDN, and the reveal path is excluded from caching by a CDN rule as well, so a wrong header cannot turn it into a cached object.
   3. Each request runs one conditional delete that returns the deleted row; only the first matches. The store applies writes to one key one after another, so exactly one request finds state = active. In SQL this is DELETE … RETURNING content; in Cassandra a lightweight transaction flips the state and only the request told it was applied returns the content it read.
   4. For a burn paste above 8 KB, the winner flips the state to burned and streams the blob. The same conditional write decides the winner; the loser finds state = burned. The blob is deleted later by the sweeper, after a 10-minute delay so a slow download can finish. If the winner’s connection dies mid-stream the content is lost: at most once, never twice.
   5. The winner gets the text with no-store; the other reader gets 410 “already read”. A 410 tells the intended recipient something useful: someone opened it first, so the secret in it should be rotated. The link holder already knows the paste existed, so the 410 leaks nothing new.
   6. The change stream carries the burn to the sweeper, which removes the leftovers. The expiry index entry and the blob are deleted by the same idempotent cleanup that handles expiry and deletes. Burn pastes are never deduplicated, so the blob has no other reader to protect.
5. Expiry and deletion in the background. Correctness comes from the read path; this flow reclaims the storage. Five million rows a day, swept from an index ordered by expiry time, with lifecycle rules as the backstop.
   1. A one-hour paste is requested at minute 61, before the sweeper has reached it. Any cached copy already lapsed at minute 60, because its lifetime was capped at the remaining life. The origin finds the row, sees expires_at in the past and answers 410 as if the row were gone.
   2. Every minute, the sweeper reads the expiry index buckets for hours that have closed. Each hour is split into 64 shards by hash of the id, so no single partition takes all of an hour’s writes, and 64 workers can drain an hour in parallel by leasing shards. The sweeper only ever touches what is due, never the 9 B rows that are not.
   3. Sweeper deletes the expired rows with a condition on expires_at. The condition makes the delete idempotent and safe against a race with an owner delete or a burn: a row that is already gone simply matches nothing. Owner and expiry index entries go in the same batch.
   4. Sweeper deletes the expired blobs in batches of up to 1,000 keys. Half a million blobs a day is about 500 batch calls. Expiring blobs are never shared, so no reference count is consulted. Lifecycle rules on exp/1d/, exp/7d/, exp/31d/ and exp/366d/ delete anything older than the class plus a day, so even a sweeper that is down for a week cannot leave bytes behind for long.
   5. An owner deletes a never-expiring public paste; the row is marked deleted and the change reaches the sweeper. The read path returns 404 as soon as the row is marked, so the delete is effective at origin immediately; everything after that is cleanup of copies.
   6. Sweeper purges the CDN by surrogate key and decrements the shared blob’s reference count. Deletes are unscheduled, unlike expiry, so they are the case that needs a purge. The blob may be shared with other public pastes of the same bytes, so the sweeper only decrements its count.
   7. A garbage collection pass deletes shared blobs whose count has been zero for 24 hours. The grace period covers a create that is about to increment the count again, and the pass re-reads the count before deleting. A weekly reconciliation recounts references from the paste rows, because a count that is too low is the one bug here that loses data.

Deep dives

The theory behind it

• Object storage and large files: Blob storage, pre-signed uploads, chunking and resumable transfer, multipart, erasure coding, storage tiers and lifecycle, plus how media pipelines are structured.
• HTTP caching and cache headers: How browsers, CDNs and proxies decide what to cache: Cache-Control directives, max-age and s-maxage, ETags and conditional requests, immutable assets with fingerprinted URLs, stale-while-revalidate, Vary, private versus shared caches, and purging.
• Trust and safety, moderation and abuse prevention: Designing defences against spam, fake accounts, fraud and harmful content: layered checks at signup, posting and payment, rules plus machine learning, review queues, hashing known-bad media, reputation, rate limits and feedback loops.

Related

• Design a URL Shortener
• Design a News Feed
• Design Dropbox
• Design Search Autocomplete
• All 83 system design interview questions
• Run this question for a friend: the interviewer kit