Design a Content Moderation System

• Check every post, comment, image and video: Spam, adult nudity, hate, harassment, graphic violence, violent extremism and known illegal imagery. Cheap checks inside the write; heavier image, video and multimodal models after the post is accepted and before it reaches a wide audience.
• Block known violating media at upload: Match against hash lists (industry lists of known child sexual abuse material and terrorist content, plus our own lists of media removed before), surviving re-encoding, resizing and small edits.
• Gate distribution on confidence: An item whose scan is pending or whose score is uncertain reaches existing followers only (no recommendations, search or trending) until it is cleared. A slow check costs reach, not safety.
• User reports: 2 M a day, folded into one case per item and per cluster of copies, weighted by how often each reporter’s past reports were upheld. Notices from trusted flaggers get priority.
• Human review queues: Ordered by expected harm, routed by policy and language, with deadlines by severity. One decision per cluster of copies, applied to all of them. Reviewer protections and quality control are part of the tool.
• Graduated actions: Remove, withhold in a country, limit distribution, age-gate, warning screen, strike, feature limits, suspension. Each comes with a notice and a statement of reasons.
• Appeals: By the author, and by reporters when content is left up. Decided by someone other than the first reviewer, and by a person when the first decision was automated. A reversal restores the content and removes the strike.
• Policy changes without model changes: Thresholds and actions are versioned per policy, surface and country, replayed over stored scores and shadowed before they take effect.
• Re-scans, legal reports and transparency: Re-scan existing content when a hash list or policy changes; file the reports the law requires (for example to NCMEC in the US); publish transparency reports from the same records.
• Out of scope: Model architectures and training code, fake-account and bot detection (used here only as reputation signals), ads review, live streams, end-to-end encrypted messages, fact-checking programmes, and reviewer workforce scheduling.

• Write-time checks (p99 < 50 ms added): They run on every one of ~120 k items a second at peak, so only cheap checks go inline: limits, reputation, hash lookups and a small text model. Anything that needs a GPU for longer runs after the ack.
• Reach before review (limited until cleared · images p95 < 1 min · video p95 < 10 min): This drives the hardest trade-off in the design: the models that catch the most are too slow and costly to run inline, so uncertain items wait at reduced reach instead of blocking the post or spreading unchecked.
• Known illegal imagery (0 views of a hash-list match): Blocked inside the write, preserved and reported where the law requires. General reviewers never see it; a specialist team handles confirmations.
• Time to action by severity (S0 < 15 min · S1 < 1 h · S2 < 24 h): Measured from the first signal (report or detection) to the action, and as views before action. Exposure is what the user feels, so it is the number reported alongside latency.
• Precision of automated actions (≥ 98 % per policy): Measured weekly from blind second reviews and appeal reversals. A policy whose automated removals fall below it routes to review instead of acting.
• Every decision recorded (100 % in the ledger · version + evidence): Appeals, strikes, statements of reasons, legal reports and transparency counts all come from one append-only record, so they cannot disagree.
• Fail to limited, not open or shut (posting up when checks are down): A moderation outage must neither stop posting nor let unscanned items into recommendations. Missing answers mean limited reach.
• Reviewer wellbeing and privacy (exposure caps enforced in the tool): Blurred, greyscale previews by default and per-shift caps on the most severe queues. Reviewers see only the personal data a decision needs, and every access is logged.

• Items written per second: ~46 k/s avg · ~120 k/s peak. 1 B posts + 3 B comments = 4 B items/day. 4 B ÷ 86 400 ≈ 46 k/s; evenings and big events run ~2.5× that, ~120 k/s. Every one passes the inline checks, so that tier is sized for the peak with a 50 ms budget.
• Inline text model at peak: ~30 GPUs. A small distilled multilingual classifier batched every few milliseconds does ~4 k short texts a second per GPU. 120 k/s ÷ 4 k = 30 GPUs at peak, ~12 on average. Hash lookups add ~6 k/s at peak, a few cores. The inline tier is cheap; its constraint is the 50 ms deadline, not money.
• GPUs for video scanning: ~350 busy · ~700 provisioned. 10 M videos × ~1 min average × ~3 GPU-seconds per minute (frames at 1 fps, speech to text, a multimodal model on flagged segments) = 30 M GPU-s/day ≈ 8,300 GPU-hours ≈ 350 GPUs busy on average, ~700 to absorb peaks. At ~$2 per GPU-hour that is ~$17 k a day. Images (100 M × 20 ms ≈ 23 GPUs) and text are small next to it: video is 0.25 % of items and ~85 % of the GPU bill.
• Hash lists in memory: ~1.6 GB raw · ~29 GB indexed. ~50 M list entries × 32 B (a 256-bit PDQ hash) = 1.6 GB. The multi-index keeps 16 chunk tables, each entry stored with its full hash for a sequential scan: 16 × 50 M × 36 B ≈ 29 GB, held in RAM on every matcher. A lookup reads 16 buckets of ~760 entries, ~0.1 ms, so RAM rather than CPU is the cost.
• Review capacity vs demand: ~2.6 M decisions/day vs ~20 M candidates. 15,000 reviewers × 5/7 on shift ≈ 11,000 a day × 6 h on queue × ~40 decisions an hour ≈ 2.6 M/day. Candidates: ~0.5 % of 4 B items land in an uncertain band (~20 M), plus 2 M reports on ~0.8 M items and ~250 k appeals. People can look at about one candidate in eight, so the queue is a ranking and the rest stay limited.
• Enforcement ledger: ~45 M rows/day · ~16 TB/year. ~20 M automated actions a day (0.5 % of items, mostly spam), ~20 M holds kept because of a score, and ~2.6 M human decisions ≈ 45 M decisions × ~1 KB = 45 GB/day, ~16 TB/year. Kept for years because strikes, appeals, legal requests and transparency reports read it. Evidence (preserved copies) is stored apart, with its own retention.
• Stored hashes for re-scans: ~600 B hashes · ~19 TB. Every upload’s hashes are stored: 100 M images + 10 M videos × ~10 keyframes = 200 M hashes/day. Over 8 years ≈ 584 B × 32 B ≈ 19 TB. A list update streams these through an index of only the new entries: one sequential pass, under an hour on a few hundred cores, instead of re-reading exabytes of media.
• Re-scoring with a new image model: ~12 days on 200 GPUs. ~10 B images were viewed in the last 30 days and carry nearly all views. 10 B × 20 ms = 200 M GPU-s ≈ 55 k GPU-hours ≈ 12 days on 200 GPUs, ~$110 k. All ~300 B stored images would take 30× longer, so backfills run by exposure, newest and most viewed first.

• Inline checks (limits · reputation · text · hashes): The cheap tier, run inside the write: rate limits scaled by account reputation (age, verified phone, prior strikes, upheld reports), text normalised (Unicode NFKC, look-alike characters, zero-width characters) and scored by a small multilingual model in ~5 ms on batched GPUs, and a hash lookup per media file. ~120 k items a second at peak on ~30 GPUs. It returns signals and scores, never actions: the policy engine decides.
• Apps (post · report · appeal): The mobile and web apps. Media goes to object storage through a presigned upload before the post is sent, and the upload pipeline computes each file’s SHA-256 and perceptual hashes, so a post carries media ids rather than bytes. The apps also file reports and appeals and show notices with the statement of reasons for any action.
• API gateway (auth · per-IP limits): Authenticates users and applies coarse per-IP and per-device limits against floods, then routes posts and comments to the content service, and reports and appeals to the review platform. It holds no moderation logic, so policy never depends on which client sent the request.
• Content service (sharded by item id · outbox): Source of truth for posts, comments and media metadata, including each media file’s stored hashes and every item’s visibility. On a write it calls the inline checks with a 50 ms deadline, then commits the item, its reach state and an item.created event in one transaction. If the checks time out, the item is accepted as limited rather than rejected, except for authors with low reputation, whose items are held until checked.
• Policy engine (versioned rules · score store): Turns calibrated per-label scores plus context (surface, country, viewer age, account signals, reports, hash matches) into a decision under an immutable, versioned policy: an action and its scope, a reach state, and whether a person must look, with severity and queue. It keeps each item’s latest scores and model versions (~150 B per item for 90 days), so a threshold change is replayed over stored scores instead of rerunning models. Models never contain thresholds.
• Hash matcher (PDQ multi-index · ~50 M in RAM): Every list of known violating media in memory: industry lists of known child sexual abuse material and terrorist content, and our own lists of media removed before, ~50 M 256-bit PDQ hashes plus SHA-256 and video signatures. Each hash is split into 16 chunks of 16 bits with one table per chunk, so a near-match within 15 bits takes ~0.1 ms and within 31 bits about a millisecond. Each list carries its own match radius and action, and hashes of near-blank images are never matched.
• Event log (Kafka · key = item id): item.created and item.edited events from the content service’s outbox, keyed by item id so an item’s versions stay in order. Scanners consume it in separate lanes for hashing, images, video and backfill, so a re-scan never delays fresh uploads. Everything slow sits behind it, so posting latency never depends on a GPU queue.
• Async scanners (GPU · image, video, OCR, multimodal): The heavy tier, after the ack and before wide reach: image models (~20 ms of GPU each), OCR so text drawn into pictures is scored, video at ~3 GPU-seconds per minute (a cheap per-frame model at 1 fps, speech to text, then a multimodal model on flagged segments), and a model over image and caption together. Video is 0.25 % of items and ~85 % of the ~400 busy GPUs. It also places each item in a near-duplicate cluster using an index of the last 7 days of uploads.
• Review platform (queues · routing · tool · QC): Folds reports and policy referrals into one case per item or cluster, weights reports by reporter reliability, and orders each (policy, language) queue by expected harm with deadlines by severity. Serves cases to reviewers certified for that policy and language in a tool with blurred, greyscale previews and per-shift exposure caps, and mixes in known-answer items and blind second reviews. Appeals are cases too, never given to anyone who decided the item before.
• Enforcement service (ledger first · idempotent effects): The only writer of enforcement. It records each decision in the ledger first, then applies its effects keyed by decision id: visibility in the content service, reach state, account standing, notices with a statement of reasons, banked hashes and legal reports. A reconciler replays the ledger to repair any effect that failed, and the service serves decision records to the review platform for appeals.
• NCMEC & regulators (reports · lists · statements): Outside bodies: NCMEC, to which US providers must report apparent child sexual abuse material they become aware of; industry hash-sharing lists; the EU Commission’s database of statements of reasons under the Digital Services Act; and authorities that send removal orders. Their rules set deadlines, preservation periods and what the ledger must be able to prove.
• Training & re-scans (labels · active learning · backfills): Turns ledger decisions into policy-versioned labels, picks the active-learning slice and the random prevalence sample for review, retrains and calibrates models, and runs re-scans: new hash list entries against ~600 B stored hashes (~19 TB, under an hour), and new models over recently viewed media first within a GPU budget. Transparency counts are computed here from the ledger.
• Enforcement ledger (append-only · by day · hash-chained): One row per enforcement decision: subject, action and scope, policy id and version, source, model versions and scores, hash list entry and distance, pseudonymous reviewer id, evidence pointer, and the parent decision for appeals and reversals. ~45 M rows a day, ~16 TB a year, never updated: a reversal is a new row. Strikes, statements of reasons, transparency counts and training labels are all computed from it.
• Reach gate (in-memory KV · per item version): Per item version: eligible, limited (followers and direct links only, with a 10 k impression cap for authors over 100 k followers), restricted (age-gated or behind a warning screen, never recommended), withheld in listed countries, or removed. Recommendation and search indexes follow its change stream, and every recommended page is checked against it in one batched read before serving. A missing or unreadable key reads as limited, so a failure costs reach, not safety.
• Feeds, search, recs (outside this design): Every surface that hands an item an audience: followers’ feeds, recommendations, search, trending and hashtag pages. They run no moderation of their own: they index only eligible items and check each page of candidates against the reach gate, dropping anything not eligible for this viewer’s country and age.

1. A photo post, checked inline and cleared: The main write path. Cheap checks run inside the request and decide whether the post is accepted and at what reach; the image models run after the ack, and the post becomes eligible for recommendations only when they clear it. The author never waits for a GPU, and nothing unscanned reaches a stranger’s feed.
   1. The app posts a photo it has already uploaded, with a caption; the gateway authenticates the author and forwards the post to the content service.
   2. The content service calls the inline checks with the author, the text and the media hashes, under a 50 ms deadline.
   3. The checks normalise and score the caption, and look up the photo’s hashes in the hash lists.
   4. The policy engine turns the scores into a write-time decision for this surface and country.
   5. The content service commits the post with its reach state and an outbox event, then acknowledges the author.
   6. The image scanners classify the photo, read any text in it, and score the photo and caption together.
   7. The scanners send the scores to the policy engine, which decides the post is clear.
   8. The enforcement service sets the post’s reach to eligible, and recommendations and search can now serve it.
2. A reported comment reaches the right reviewer: The main human path: a hateful comment in Portuguese under a popular post is reported 40 times. The reports become one case, ordered by expected harm, routed to a reviewer certified for hate speech in Portuguese, decided once, and recorded before anything changes.
   1. Users report the comment; the gateway sends each report to the review platform, which folds them into one case.
   2. The review platform gets the comment’s stored scores and the policy’s view of severity and queue.
   3. The case is placed in the Portuguese hate-speech queue, ordered by expected harm.
   4. A reviewer certified for hate speech in Portuguese opens the case in the review tool.
   5. The reviewer decides it violates the hate policy, and the verdict goes to the enforcement service.
   6. The enforcement service records the decision in the ledger before applying anything.
   7. It removes the comment, records a strike against the author, and sends a notice with a statement of reasons.
   8. The decision flows from the ledger to training as a label tagged with its policy version.
3. One violating video, 100,000 copies: The scale-breaking case: a graphic video is uploaded, then re-uploaded by thousands of accounts, some edited to evade matching. Reviewing each copy would take more reviewer hours than exist. The design reviews it once, applies the decision to every copy, and blocks future copies at upload.
   1. A new video passes the inline checks, is accepted at limited reach and queued for the video scanners.
   2. The video scanners score its frames, speech and segments, and find likely graphic violence.
   3. The policy engine keeps every copy limited behind a warning screen and sends the cluster to review at top priority.
   4. A specialist reviewer decides the cluster once: it violates the graphic violence policy.
   5. The enforcement service records a decision for every copy, removes them all, and banks the video’s hashes.
   6. New uploads of the video now match the banked hashes inside the write and are blocked before anyone sees them.
   7. Edited copies that miss the tight radius join the cluster through the wider radius or the embedding, and go to review as members.
4. A wrong removal, appealed and reversed: The failure path: the image model removes a breast cancer awareness post as adult nudity. The author appeals; a person who was not part of the first decision reviews it, the reversal is a new ledger row, and everything derived from the first decision (visibility, reach, the strike, the counts, the label) follows it.
   1. The image model scores a post 0.985 for adult nudity, above the automatic removal line, and the post is removed.
   2. The author gets a notice with the statement of reasons and appeals; the appeal reaches the review platform.
   3. The review platform builds an appeal case from the decision record and excludes everyone who decided the item before.
   4. A senior reviewer finds the post allowed under the health-awareness exception and reverses the decision.
   5. The enforcement service appends a reversal row, then restores the post, its reach and the author’s standing.
   6. The reversal becomes a high-weight training label, and repeated reversals in one context flag the threshold.
5. A hash list update reaches old content: The background path: an industry list adds 20,000 hashes of known illegal imagery. Copies already on the platform must be found without re-reading exabytes of media; the hashes stored at upload turn it into one pass over ~19 TB. Matches are removed, preserved and reported, and the reports and counts come from the ledger.
   1. The enforcement service pulls a list update from an industry partner and loads the new entries into the hash matcher.
   2. The re-scan job streams every stored media hash through an index built from only the new entries.
   3. Candidates are confirmed against the original files and the policy engine decides.
   4. The enforcement service records each match, removes the media everywhere, and preserves a copy for the report.
   5. The enforcement service files the report the law requires and records that it did.
   6. Each period, the transparency job counts decisions from the ledger by policy, source and outcome.

Ask: Why not run your best models on every post before it is published? And if you don’t, what stops a bad video spreading while it waits?

Good answers name: Tiers by cost plus reach states: inline checks decide acceptance and reach, heavy models run after the ack, uncertain items stay limited until cleared, Run every model inline before publishing, Publish fully, scan afterwards, take down what fails, Hold items invisible until every check passes.

Our pick: Three tiers by cost: inline (limits and reputation, exact and perceptual hash lookups, normalised text through a small multilingual model; ~20 ms typical, 50 ms deadline), async (image, OCR, multimodal and video models; images p95 under a minute, video under ten), and human review. Every item version gets a reach state: eligible (any surface), limited (followers and direct links, no recommendations, search or trending, and a 10 k impression cap for authors with more than 100 k followers until cleared), restricted (age-gated or behind a warning screen, never recommended), withheld in listed countries, or removed. Text from an author in good standing with clean inline scores is eligible at once; media is limited until its scan clears, and anything in an uncertain band stays limited until a person clears it. Surfaces check the gate in one batched read per page and treat a missing key as limited, so a slow, failed or lagging scan can only cost reach. The TikTok question on this site uses a staged audience for a related reason; here the gate is driven by uncertainty about policy, not by how well the item performs.

1. The video scanners fall an hour behind. What happens?
   New videos stay limited longer: followers can watch, recommendations cannot. Nothing unscanned leaks out and posting is unaffected. The backlog drains by expected reach rather than first in, first out, so a video from a 5 M-follower account that is gaining views is scanned before a clip with 20 views, and the hash lane has its own pool so known material is still caught in seconds. The alert is scan lag by reach band, and the cost of the incident is the views that waited.
2. An account with 80 M followers posts a video. Isn’t followers only already wide distribution?
   Yes, which is why limited means an impression cap for large accounts: 10 k until cleared. Their uploads also go to the front of the scan queue, so the cap rarely bites for long. Reputation lowers their prior but never skips the scan, and if their video lands in the uncertain band it goes to review near the top, because its predicted reach is enormous.
3. Someone posts a harmless photo, waits for it to be cleared, then edits it.
   An edit is a new version, and a new version goes back through the inline checks and back to limited until the scanners clear it. The reach gate is keyed by item and version, so an eligible state cannot carry over to new content. Swapping media, changing a caption and adding a link all count as edits.
4. The reach gate is unavailable. Do surfaces fail open?
   No. A key that cannot be read counts as limited, so recommendations and search drop the candidates they cannot verify and fill the page with others. Followers’ feeds keep working, because limited items are allowed there anyway. What the outage costs is freshness in recommended content, which is the right side to fail on.
5. How do you know gating isn’t costing creators too much?
   Measure it: time from post to eligible by media type and percentile, the share of items still limited after an hour, and the views those items would have had, estimated from cleared posts by similar authors. Put that next to views before action for items later removed. Where a segment pays a large hold for little catch, such as text from long-standing accounts, the policy makes it eligible at once.

Ask: Why not just keep a set of SHA-256 hashes of known violating images? And when a list gains 20,000 entries, how do you find the copies already on the platform?

Good answers name: Perceptual hashes (PDQ for images, TMK+PDQF for video) with per-list radii, a copy-detection embedding for heavy edits, and every upload’s hashes stored for re-scans, Cryptographic hashes only (SHA-256 or MD5), Learned embeddings with nearest-neighbour search only, Re-download and re-hash all stored media on each list update.

Our pick: At upload the media pipeline computes SHA-256, PDQ with its quality score and the mirrored and rotated variants, and for video a TMK+PDQF signature plus PDQ on ~10 keyframes, and stores them with the item. The matcher holds ~50 M list entries in RAM as a multi-index: each 256-bit hash split into 16 chunks of 16 bits, one table per chunk. Two hashes within 15 bits must agree exactly on at least one chunk, so a lookup is 16 bucket reads and ~12 k candidate comparisons, about 0.1 ms; out to 31 bits, probing each chunk’s one-bit neighbours is ~17× the work, about a millisecond. Each list carries its own policy: an industry list of known illegal imagery blocks within 15 bits and sends 16 to 31 to a specialist; our own lists of previously removed media route to review, because context decides whether a repost violates. Hashes below a quality score are never matched. A list update loads into the matcher within a minute, then a re-scan streams all ~600 B stored hashes through an index of just the new entries, under an hour on a few hundred cores, and confirms candidates from the original file before acting.

1. Why not set a wide threshold and catch every edited copy?
   Because false matches grow fast with the radius, and on a list of known illegal imagery a false match means a removal, a disabled account and possibly a report about an innocent person. So the tight radius acts and the wide radius only routes to review, and the false-match rate is measured per list and per distance band from reviewer outcomes. The wide band is where the copy-detection embedding earns its cost, as a second opinion before a person looks.
2. Someone crafts an innocent image that collides with a listed hash.
   That has been shown publicly: researchers produced collisions for Apple’s NeuralHash within weeks of its release in 2021, and any perceptual hash can be attacked that way. A crafted collision does not look like the listed image, so a hash match is evidence, not proof: account-level actions on a list match need a second signal (the image model, the embedding) or a specialist. A flood of collisions also shows up as one list entry matching thousands of unrelated uploads, which pages the list owner.
3. A better hash algorithm comes out. Now what?
   Then stored hashes do not help and the re-scan has to read media. Compute the new hash for every upload from now on, backfill media viewed in the last 30 days first (a few percent of storage, nearly all of the views), and work through the long tail over months within a fixed budget. Shared lists move to a new algorithm slowly, so both are matched side by side for a long time.
4. How do you catch spam text disguised with look-alike characters and spacing?
   Normalise before anything scores it: Unicode NFKC, look-alike characters mapped to one form (the Unicode confusables data exists for this), zero-width characters removed, repeated letters collapsed, links resolved to their final domain. Then near-duplicate detection on the normalised text, MinHash or SimHash, clusters a spam wave that varies a few words per copy, so one decision covers it. Text drawn into images goes through OCR into the same path.
5. One upload matches a list of known child sexual abuse material. Do you disable the account at once?
   For a tight, high-quality match, policy treats it as severe enough to disable, and the design supports that. Weaker evidence (the wide radius, a low-quality hash, a single match on an account with a long clean history) goes to a specialist first. For our own lists of previously removed media, never automatically: those are reposts of content that violated in its original context, and a strike needs a person or a confident model on this copy.

Ask: A new law in one country requires a category of content to be removed there within 24 hours. Do you retrain the model?

Good answers name: Calibrated per-label scores in; a versioned, declarative policy evaluated per surface, country and viewer; actions out, Models output actions directly, Thresholds written into each consuming service, A separate model per language or region.

Our pick: Every model emits calibrated probabilities per label, such as hate.dehumanising 0.91, with its version, and the engine stores each item’s latest scores (~150 B per item for 90 days). The policy is data: rules matching on labels and score bands, surface, country, viewer age, account signals and detection source, each returning an action (remove, withhold, restrict, age-gate, warning screen, limit, review with a severity, or none), a scope (global or a list of countries) and a queue. Rules run in a fixed order: legal orders and hash-list rules, then country-specific legal rules, then global policy with its exceptions, then defaults; within a scope the most restrictive result wins. A version is an immutable snapshot, changes need a second approver, and each change is replayed over the last 7 days of stored scores (items hit per action, reviews created) and shadowed for a day before rolling out surface by surface. For the new law: a rule scoped to that country with action withhold-in-country and a 24-hour review deadline, plus a replay over stored scores of items currently being viewed there, so existing content is caught without touching a model.

1. You ship a new hate model and its scores shift. Do all the thresholds break?
   Not if every model is calibrated per label before it ships, so that 0.9 means about 90 % of such items violate, measured on fresh labelled sets in the major languages. The rollout compares decision rates per rule under the old and new model on the same items in shadow, and any rule whose hit rate moves by more than a few percent is reviewed before the switch. Rules refer to labels, never to model versions.
2. Two policies disagree: one removes nudity, another allows it for health awareness.
   Exceptions are rules with explicit precedence: an allow rule for a recognised context (health, art, news) runs before the generic remove rule, and its condition is a context label from a model or a reviewer. When the context model is not confident, the result is review rather than removal, because those are exactly the cases automated removal gets wrong.
3. Content is legal in one country and illegal in another.
   Scope is part of the action: the rule withholds the item in the countries where it is illegal and leaves it eligible elsewhere, and the reach gate is read with the viewer’s country. The ledger records the scope and the legal basis, and the notice tells the author where it is withheld and why. A global removal can only come from global policy.
4. An authority sends a removal order with a one-hour deadline.
   That is a source, not a model. The order arrives through a legal intake, maps to a rule with an action, a scope and a deadline, and lands in an S0 queue for legal review. The EU regulation on terrorist content online, for example, gives hosting providers one hour from a competent authority’s removal order, so the deadline drives priority and paging. Every order and its outcome go into the ledger, because transparency reports count them separately.
5. How do you test a policy change?
   Three steps: a suite of known items per rule that must produce the expected action, a replay over 7 days of stored scores reporting hits and reviews created by country and surface, and a day of shadow evaluation on live traffic where decisions are logged but not applied. Reviews created matters most: a change that adds 2 M cases a day silently breaks every queue’s deadlines.

Ask: Your reviewers can make 2.6 M decisions a day and 20 M items want one. Which items do people see, and how do you keep the decisions good?

Good answers name: Expected-harm priority with severity deadlines, routing by policy and language, one case per cluster, and quality control built in, First in, first out per queue, Order by number of reports, Automate everything, no people.

Our pick: Each case gets priority = P(violation) × severity weight (S0 1,000, S1 100, S2 10, S3 1) × predicted views over the next 24 hours, from current view velocity, the author’s audience and the cluster’s total, recomputed every few minutes, and a deadline by severity (S0 15 min, S1 1 h, S2 24 h) that takes over as it approaches. Queues are per (policy family, language), reviewers are certified per policy and language, S0 and S1 queues in the main languages are staffed around the clock by sites in different time zones, and rare languages use machine translation for triage with a native speaker for the final call. Cases are per cluster (media by perceptual hash and embedding, text by MinHash on normalised text), and a verdict fans out to every member. The tool shows media blurred and greyscale by default, video as a strip of keyframes with the flagged segments, audio off until asked, and enforces per-shift caps on S0 and S1 items. About 3 % of each queue is known-answer items, about 5 % of decisions get a blind second review, and removing an item with over 1 M views or disabling an account with a large audience needs two reviewers to agree.

1. Fake views push a borderline item up the queue, or hold another one down.
   Predicted reach counts views from accounts with normal histories and discounts bursts from new accounts on one network, the same signals the abuse pipeline uses. Pushing an item up the queue only gets it reviewed sooner, which is harmless. Holding one down needs it to look quiet, and a quiet limited item is not spreading, so the attack mostly delays a review that was not urgent.
2. Two reviewers disagree on 15 % of hate speech cases. What do you do?
   Treat it as a policy problem before a people problem. Disagreement concentrated on one rule or one language means the guideline is ambiguous: policy writes examples for the disputed cases and adds them to the known-answer set. Individual reviewers far below their peers on known-answer items get coaching and leave that queue until recalibrated. Disputed cases go to training with lower weight or a third opinion.
3. How do you protect the reviewers who see the worst content?
   Reduce exposure, then cap it. Hash matches on known illegal imagery are never shown to general reviewers; a specialist team works those queues with tighter limits. Previews are blurred and greyscale until the reviewer chooses to look, video is a strip of frames rather than playback, audio is off, and the tool counts each reviewer’s S0 and S1 items per shift and per week and stops serving them past the cap. Breaks, rotation and counselling are policies; the cap is enforced in code.
4. A coordinated group mass-reports an activist’s posts. Does volume decide?
   No. Reports are weights, not votes: each reporter’s weight comes from how often their past reports were upheld, and reports from recently created or linked accounts count for little. Volume can raise priority, so a person looks sooner, but no number of reports removes content without a model score or a reviewer. Accounts that keep filing rejected reports lose weight, and the DSA expects platforms to act against repeated manifestly unfounded notices.
5. How do you know review is working?
   Time to action by severity against deadlines, views before action for removed items, accuracy on known-answer items, second-review agreement and appeal reversal rate per policy and language. Above all prevalence: the share of a random sample of all views that reviewers label violating. Prevalence is the one number that says whether the whole system works, because it counts what nobody reported and no model flagged.

Ask: Can you train your classifiers on reviewer decisions? Reviewers only ever see what the models and reporters already surfaced.

Good answers name: Policy-versioned labels from every decision, weighted by agreement, plus an active-learning slice and a random prevalence sample, Train on every reviewer decision as it arrives, Expert-labelled sets only, User reports as labels.

Our pick: Every decision in the ledger becomes a label event with its rule and policy version, the reviewer’s certification, whether it was a second review or an appeal, and whether it was later reversed; appeal reversals of automated decisions are the most valuable labels there are and are weighted up. About 5 % of review capacity, ~130 k items a day, goes to items chosen for label value: scores in a band around each threshold, items where two model versions disagree, new clusters, and languages with thin data; they are reviewed like any case and their actions apply. Another ~20 k random views a day, stratified by surface and country, are labelled for prevalence, which gives unbiased recall and the headline number. When a definition changes, the evaluation set is relabelled under the new version first, and training drops older labels for the changed rule until enough new ones exist. Models retrain monthly; a confirmed new evasion goes into hash lists, link blocks or a text rule within hours rather than waiting for the next model.

1. Isn’t sending low-harm items to review a waste when deadlines are being missed?
   The slice is a budget and the first thing to shrink in a surge: S0 and S1 deadlines come first. In normal operation it pays for itself: a model one point more precise at a threshold that removes 100 k items a day avoids about 1,000 wrong removals, and their appeals, every day.
2. How do you measure recall when you don’t know what you missed?
   From the random sample. Label a random sample of views, count the violating ones, and check what share the system had already caught. That is recall measured on views, which weights each miss by its exposure: a viral miss counts once per view, as it should.
3. One review site applies a policy inconsistently. Do its labels poison the model?
   They would, which is why labels carry the reviewer’s site and certification and are weighted by measured accuracy on known-answer items for that queue. A site whose agreement with the known answers drops shows up in the quality numbers within days, its labels for that period are down-weighted, and the cases with the largest disagreement are reviewed again.
4. Spammers change tactics every few days. Does a monthly model keep up?
   Not on its own, and it does not need to. The fast loop is not training: a decision on a new spam cluster blocks its link domains, banks its media hashes and adds normalised text patterns within the hour, while account and network signals catch the accounts behind it. The next model learns the pattern, and meanwhile the near-threshold slice concentrates labels on what is new.

Ask: A year later a user asks why their account was suspended, a regulator asks how many removals were automated, and an appeal reverses one of the strikes. Where does each answer come from?

Good answers name: Append-only ledger written first; visibility, reach, standing, notices and reports applied as idempotent projections, Status columns on the post and a strike counter on the account, Reconstruct decisions from application logs, Mutable status plus a separate audit log.

Our pick: The enforcement service writes each decision to the ledger before applying it: decision id, subject (item, cluster member or account), action and scope, rule and policy version, source (hash list and version, model, reviewer, appeal, authority order, re-scan), model versions and scores, list distance, pseudonymous reviewer id, an evidence pointer, the parent decision for appeals and reversals, and timestamps. It is partitioned by day, indexed by item and account, and hash-chained per partition as the audit-logging guide describes. Effects are projections keyed by decision id, so they are idempotent: visibility, reach, standing, notices with statements of reasons, NCMEC reports, and the statements of reasons the DSA asks platforms to send to the Commission’s database; a reconciler replays the last day to repair anything that did not apply. Standing is computed from upheld decisions, a strike per violation weighted by severity over 90 days, with a ladder from warning to feature limits to suspension, and an S0 violation disabling at once; a reversal removes its strike and standing is recomputed. Transparency counts are queries over the same rows by decision time, so the report and every individual answer agree.

1. The ledger write succeeds but the removal fails. Is the post still up?
   For seconds at most. Every effect is keyed by the decision id and retried until it applies, and the reconciler compares each projection with the ledger and fixes the difference, so the post comes down on the retry and the notice is sent once. The reverse order, remove first and record later, is the one to avoid: a removal with no record cannot be explained, appealed or counted.
2. Someone appeals a decision made under a policy that has since changed.
   The ledger has the version that applied, so the reviewer sees both the rule then and the rule now. If the action was right under the old policy but the new one allows the content, it is restored and the strike removed, because nobody should carry a penalty for something now allowed. If it was wrong under the old policy, it is reversed as an error, which also counts against the model or reviewer that made it.
3. How do you make sure an appeal goes to someone else?
   The appeal case carries every decision id in the chain, and the router excludes every reviewer in it, preferring another site and a more senior certification. For automated decisions, the first person to see the item decides. Reversal rates are tracked per original reviewer and per model version: for a reviewer it is a quality signal, for a model it is a threshold problem.
4. A user deletes their account. Must the ledger forget them?
   It must stop identifying them wherever the law allows. Their content is deleted with the account; ledger rows keep pseudonymous subject ids so counts and other people’s appeals still work, and the mapping from those ids to the person is deleted. Evidence preserved for a legal duty, such as a report to NCMEC, is kept for the period the law sets under restricted access, then crypto-shredded. The privacy guide covers the general pattern.
5. What goes into a transparency report?
   Per policy and period: items actioned by action and by detection source (found by us before any report, or reported), appeals filed, items restored on appeal and restored without one, authority orders and their outcomes, median time to action, and prevalence. All are ledger queries dated by decision time and reproducible later, and the same rows produce each user’s statement of reasons, so the totals and the individual answers cannot disagree.