Design GitHub

Host 400 M git repositories on replicated file servers, serve 100 k fetches a second, and run pull requests, forks and webhooks on top.

Last updated 2026-10-26. Difficulty: hard. Patterns: git, replication, quorum, content-addressing, thundering-herd. Reported at Microsoft, Atlassian, Google, Meta, Amazon, Stripe.

Functional requirements

• Clone, fetch and push over HTTPS and SSH. Standard git clients speaking the smart protocol, version 2 where the client supports it. Authentication by token or SSH key, branch protection enforced on push. Anything that works against git itself must work against us.
• Browse code and history in the web UI. Files at any commit, history, blame and compares, read straight from the repository. A popular README is viewed thousands of times a minute, so rendered output is cached by object id.
• Pull requests. Propose merging a branch, often from a fork, into another: a diff, line comments, reviews, required checks, and merge, squash or rebase. Whether it merges cleanly is shown before anyone presses the button.
• Forks. One click and instant, even for a 5 GB repository, and a popular repository has tens of thousands of them. A fork must not copy its parent’s history.
• Webhooks and CI triggers. Every push, pull request and review becomes an event delivered to subscribed endpoints and CI systems within seconds, signed, at least once.
• Code search. Substring and regex search across every repository the user can read: all public ones plus the private ones they have access to, and nothing else.
• Large files. Models, videos and game assets through Git LFS: git stores a small pointer and the bytes go to object storage. Plain git objects over 100 MB are refused.
• Out of scope. Issues and project boards (plain rows in the metadata database), the CI runners and deployments themselves (see the Code Deployment System), packages, Pages and Copilot. Webhook delivery uses the standard machinery from the webhooks guide; here we cover what is specific to git.

Non-functional requirements

• No acknowledged push is ever lost (2 of 3 replicas before ack). A push that printed “main -> main” and then vanished in a failover breaks CI, deploys and trust. This is the requirement that drives the hardest trade-off: synchronous writes to a quorum cost push latency, in exchange for never losing or reordering a ref.
• Refs never go backwards (linearizable per ref). Once anyone has seen main at commit X, no replica may serve an older main. CI, deploys and every cache key on the commit a ref points to.
• Git availability (99.95 % reads and writes). Reads survive losing two of a repository’s three replicas and writes survive losing one. A dead file server must not take a single repository offline.
• Latency (no-op fetch < 100 ms · push ack p99 < 1 s). No-op fetches are what CI pollers and IDEs do all day, so they must be cheap for us as well as fast for them: one round trip and a short ref list. Push latency is dominated by the replica vote, about four round trips.
• Scale (400 M repos · ~100 k git ops/s peak). Clones and fetches outnumber pushes about 100 to 1, and CI and automation make most of them. Repositories range from a few KB to monorepos of 100 GB.
• Hot repositories stay contained (one repo cannot starve its neighbours). A repository cloned a million times a day, or fetched by 2,000 CI jobs in one minute, shares its three file servers with millions of other replicas. Its load must land on caches, not on its replicas.
• Private means private (zero cross-repository reads). Private contents never leak through forks, caches, code search or webhooks. Every git request, cached pack and search hit is checked against the caller’s permission on that repository.
• Event freshness (webhook p95 < 10 s after push). CI that starts a minute late is a minute added to every developer’s loop. At least once and signed; order is not guaranteed.

Back-of-envelope estimates

• Git operations per second: ~35 k/s avg · ~100 k/s peak. Assume 3 B clones and fetches a day, about 60 % from CI and automation: 3 × 10⁹ ÷ 86 400 ≈ 35 k/s. US and European working hours overlap into a peak of about 3× ≈ 100 k/s.
• Pushes per second: ~350/s avg · ~1 k/s peak. 30 M pushes a day ÷ 86 400 ≈ 350/s, about 1 k/s at peak, so reads outnumber writes ~100 : 1. Each push is a few round trips to three replicas: ~3 k replica writes a second over 1,500 servers. Write throughput is not the problem; write consistency is.
• Repository sizes and storage: ~10 PB unique · ~30 PB on disk. Heavy-tailed: median under 1 MB, p99 around 1 GB, a few hundred monorepos of 10–100 GB. Assume a mean of 25 MB once forks share their parent’s objects: 400 M × 25 MB = 10 PB, × 3 replicas = 30 PB, growing roughly 30 % a year.
• File servers: ~1,500 servers · ~800 k replicas each. NVMe servers with ~30 TB each, kept ~70 % full to leave room for repacks and moves: ~20 TB of repositories per server. 30 PB ÷ 20 TB = 1,500, and 400 M × 3 ÷ 1,500 = 800 k replicas per server, almost all tiny and idle. A server’s load comes from its few busy ones.
• CPU to serve git at peak: ~8 k cores. Assume 97 % of operations are incremental fetches at ~20 ms of CPU (ref list, negotiation, a small pack) and 3 % are clones at ~2 CPU-s with bitmaps: 97 k × 0.02 + 3 k × 2 ≈ 8 k cores, about 17 % of 1,500 × 32. Without reachability bitmaps the server side of a kernel clone took 44 s instead of 1.6 s, so clones alone would need ~130 k cores: three times the fleet.
• One hot repository: ~70 Gbit/s at peak. 1 M full clones a day of a 250 MB pack, peaking at 3×: 1 M × 250 MB × 3 ÷ 86 400 ≈ 8.7 GB/s ≈ 70 Gbit/s, plus ~70 cores of pack CPU. Its three file servers have 75 Gbit/s and 96 cores between them, shared with ~2.4 M other replicas. One repository can saturate its hosts, so its clones must be served from somewhere else.
• CI herd after one push: ~130 cores · ~13 GB/s for a minute. A push to a busy monorepo starts 2,000 jobs that each shallow-clone the new commit within 60 s, a 400 MB pack costing ~4 CPU-s to build: 2,000 × 4 ÷ 60 ≈ 133 cores, and 2,000 × 400 MB ÷ 60 ≈ 13 GB/s. Every one of those requests is identical, so building the pack once turns 8,000 CPU-seconds into 4.
• Webhook deliveries: ~6 k/s avg · ~20 k/s peak. 30 M pushes plus ~170 M pull request, review, comment and check events = 200 M events a day, each delivered to ~2.5 endpoints (repository hooks, organisation hooks, installed apps): 500 M a day ≈ 5.8 k/s, bursting to ~20 k/s. Each push delivery to a CI system turns into fetches, which is how webhooks become the CI herd.

Components

• LFS & bundle store (S3-class · behind a CDN): Holds Git LFS objects keyed by their SHA-256, and pre-built clone bundles of hot repositories. Clients fetch both from the CDN with signed URLs, so gigabyte binaries and a million full clones a day never touch a file server. These are large, immutable, read whole and never deltified: exactly what object storage is good at, and exactly what git objects are not.
• Developer (git CLI · browser · IDE): Runs standard git over SSH or HTTPS and uses the web UI and API for pull requests and browsing. The client does real work: it negotiates which objects it already has, verifies every object it receives against its hash, and with shallow or partial clone decides how much history it wants.
• CI runners (Actions · Jenkins · Buildkite): Receive push and pull request webhooks and fetch the new commit, thousands of jobs at once for a busy repository. Ephemeral runners make shallow clones of exactly that commit, so their requests are byte-identical, which is what makes them cacheable. Most git reads come from machines like these, not from people.
• Web & API (Rails · REST and GraphQL): The UI and API for everything that is not the git protocol: pull requests, reviews, forks, settings, search. Repository contents come from the file servers over an internal RPC (tree at a commit, blob, diff, merge), never by cloning. Rendered blobs and diffs are cached by object id, which can never go stale because an object never changes.
• Git frontends (SSH · smart HTTP · auth): Terminate SSH and HTTPS, authenticate the key or token, check the caller’s permission on the repository and its branch protection rules, then hand the git stream to the router. Stateless. They also cap concurrent pack builds per repository and per user, the last line of defence when one repository is hammered.
• Webhook delivery (per-endpoint queues · HMAC): Turns each event into one delivery per subscribed endpoint (repository hooks, organisation hooks, installed apps), signs it with that endpoint’s secret and POSTs it with a 10 s timeout. Retries back off per endpoint, so one dead CI server never delays anyone else. About 6 k deliveries a second on average.
• Metadata DB (MySQL · partitioned by domain): Users, permissions, repositories and their fork networks, pull requests, reviews, comments, check results and webhook subscriptions. Nothing from inside git lives here except the commit ids that rows point at. GitHub has written that it runs on 1,200+ MySQL hosts in 50+ clusters, split by feature domain (2023).
• Repo router (routes · votes · repair): Knows where every repository lives (repository → fork network → three file servers) and which replicas are in sync. Reads go to the least-loaded in-sync replica; pushes are streamed to all three and committed when two agree on the ref update. It also owns replica health: it marks replicas out of sync, re-replicates networks that fell below three, and moves networks to rebalance disk and CPU.
• Pack cache (keyed by request · single-flight): Generated packs keyed by a hash of (repository, wants, haves, depth or filter, capabilities), on a fleet of cache servers with NVMe and fast NICs. Concurrent identical requests wait for the one being built and stream it as it is written. It takes both the CPU and the bytes of a CI herd off a repository’s three replicas; an entry is only ever served for the repository that produced it.
• File servers (git on local NVMe · ×3): About 1,500 servers, each holding ~800 k repository replicas as ordinary bare git repositories on local disk, grouped by fork network. They run git itself: receive-pack for pushes, upload-pack and pack-objects for fetches, merge-tree and diff for the web. Each also repacks its busy repositories continuously (geometric repacks, multi-pack indexes, bitmaps), which is what keeps fetches cheap.
• Event bus (Kafka · keyed by repository): Ref-update events (repository, ref, old, new, pusher, push id) relayed from the replicas’ committed ref journals, plus pull request and review events from the web tier. Keyed by repository so one repository’s events stay in order. Webhooks, PR workers, code search and notifications each consume it at their own pace.
• PR workers (background merges · diffs): When a pull request’s head or base moves, ask a file server for the test merge (git merge-tree with merge-ort, no working copy), store it as refs/pull/N/merge and record mergeable or conflicting in MySQL. Debounced and prioritised by who is looking, because one push to a busy main invalidates thousands of open pull requests at once.
• Code search (ngram index · sharded by blob id): A separate index of default-branch contents, fed by push events and reading only the changed blobs. Documents are keyed by blob id, so a file shared by 10,000 forks is indexed once with a list of repositories. Each query carries the set of repositories the caller may read and is filtered inside the index. GitHub’s Blackbird shards by blob id for exactly this reason.

User flows

1. A developer pushes to main. The core write. Objects are the easy part because they are content-addressed and can be written anywhere; the only thing that needs agreement is the ref moving from old to new, and two of three replicas must agree before the client hears “ok”.
   1. If the commits include LFS-tracked files, the client uploads those bytes to object storage before git sends anything. The git-lfs pre-push hook asks the batch API which objects are missing and uploads them with presigned URLs straight to object storage. Git then pushes only ~130-byte pointer files, so the repository, its clones and its repacks stay small no matter how many 2 GB assets it tracks.
   2. The client opens receive-pack over SSH; the git frontend authenticates the key and checks write permission. Keys, tokens and permissions are read from MySQL replicas and cached for seconds. Branch protection (no force pushes to main, required reviews) can only be judged once the update commands arrive, so the frontend passes the rules along and they are enforced in the prepare phase.
   3. The router looks up the repository’s replica set and opens a receive session on each of its three file servers. acme/api → network 5512 → fs-0412, fs-1093, fs-0207, all in sync. A replica already marked out of sync is left out and repaired later; if fewer than two in-sync replicas are reachable, the push is refused rather than accepted without a quorum.
   4. The client sends its ref update commands and a packfile; the router streams the same bytes to all three replicas, which verify the objects into quarantine. Each replica runs index-pack, which checks that every object hashes to its name, resolves deltas against objects it already has, and fscks what is new. Quarantined objects are invisible to readers until the push commits, and a failed push leaves nothing behind. Writing objects is idempotent: a replica that already had some of them is unaffected.
   5. Prepare: each replica locks the ref, checks it still points at the old value, and votes with a hash of the exact update. This is git’s own reference transaction (update-ref --stdin with prepare), with the reference-transaction hook reporting what is about to change. If main moved since the client read it, the old-value check fails identically on every replica and the push is rejected as non-fast-forward, which is ordinary git behaviour.
   6. With two matching votes the router tells the replicas to commit; a replica that disagreed or timed out is marked out of sync for this repository. Commit moves the ref and publishes the quarantined objects together on each replica. Each replica then reports a checksum over all its (ref, value) pairs, updated by XOR on every change, so equal checksums mean identical refs. The out-of-sync mark is written before the client is acknowledged, so no reader can be sent to a replica that missed the push.
   7. The client sees “main -> main”, and the committed update is relayed to the event bus. Each replica appends the update to a local ref journal as part of the commit, and a relay on the replica publishes from that journal. A router crash right after the ack cannot lose the event. All three replicas may publish the same push, so consumers dedupe on push id.
2. An everyday fetch, and a file in the browser. The core read, and almost all of the load. A fetch is a negotiation followed by pack generation; protocol v2 keeps the negotiation small, and bitmaps plus reuse of stored deltas keep the pack cheap.
   1. The client connects with protocol v2 and asks only for the refs it cares about. In the original protocol the server opens by advertising every ref, and a busy repository has 500,000 of them (each pull request adds refs/pull/N/head and /merge): tens of MB on every no-op fetch. In v2 the client sends ls-refs with prefixes such as refs/heads/ and refs/tags/, and only those come back. Read permission on a private repository is checked first.
   2. The router sends the request to the least-loaded in-sync replica. Any in-sync replica can serve a read, so a repository has three times the read capacity of a single copy. “In sync” means the replica’s ref checksum matches the last committed one; a replica that missed a push would otherwise show a main that had gone backwards.
   3. Client and server negotiate: the client names the commits it wants and the commits it already has. An everyday fetch has yesterday’s main, so one round of haves finds the common base and the answer is a few new commits. A fresh clone has no haves; a shallow clone says deepen 1, and a blobless clone says filter blob:none and fetches file contents later, on demand.
   4. The file server works out which objects to send from reachability bitmaps, and reuses deltas already stored in its packs. Counting objects by walking every commit and tree is the expensive part. A bitmap per selected commit records which objects it can reach, so “reachable from wants minus reachable from haves” is a few bitwise operations. Objects already stored as deltas against something the client will also receive are copied out verbatim, with no recompression.
   5. The pack streams back; the client verifies every object against its hash and updates its remote-tracking refs. Because objects are named by their content, corruption anywhere on the path is detected by the client, not trusted. An everyday fetch of a busy repository is a few hundred KB and about 20 ms of server CPU.
   6. In the browser, viewing a file at a commit is an RPC to the same replicas, cached by blob id. The web tier resolves the branch to a commit (one ref read), walks the path through the trees, and reads the blob. The rendered HTML is cached under the blob id plus the renderer version, with no invalidation ever, because a blob id names its content forever. The only thing that changes is which commit a branch points to.
3. A push triggers 2,000 CI jobs. The scale-breaking case. Within a minute of one push to a busy monorepo, 2,000 runners ask for the same new commit; served naively that needs more CPU and bandwidth than the repository’s three file servers have. The requests are identical, which is what makes them cheap.
   1. Webhook delivery sends the signed push event to the CI system’s endpoint. Deliveries are queued per endpoint with a 10 s timeout and retried with backoff, so a slow CI endpoint only delays its own events. The receiver verifies the HMAC and dedupes on the delivery id, as the webhooks guide describes.
   2. The CI system starts 2,000 jobs, and each fresh runner asks for a shallow clone of exactly that commit. Ephemeral runners start empty, so every request is the same: want 7b21…, depth 1, no haves. 2,000 of them in 60 s, each a 400 MB pack costing ~4 CPU-seconds, would need ~130 cores and 13 GB/s from three servers that have 96 cores and ~9 GB/s between them, and that serve millions of other replicas.
   3. The router turns each request into a pack cache key and looks it up. The key hashes (repository id, wants, haves, depth, filter, and the capabilities that change the bytes). The repository id is part of the key, so an entry is never served for another repository, even one in the same fork network. The first request misses and takes a build lease; the other 1,999 find the build in progress.
   4. One in-sync replica builds the pack once, and the router tees it into the cache as it streams. Waiters read the cache file as it grows, so the second runner receives bytes almost as soon as the first. 8,000 CPU-seconds of pack building become about 4. An entry can never be stale, because the request names exact commits; entries are evicted by space and age.
   5. Cache servers stream the pack to the remaining runners, spreading the bytes over the cache fleet instead of the three replicas. A hot entry is copied to several cache servers, so no single NIC has to carry 13 GB/s. The replicas spent one build and keep serving their other repositories at normal latency.
   6. Runners that kept a workspace fetch only the new commits, and those requests collapse into a few dozen cache keys. A persistent runner has some recent main as its have, and there are only a handful of distinct recent mains, so hundreds of runners share each key. For monorepos we document blobless clones with sparse checkout as the CI default, because the cheapest pack is the one nobody asks for.
4. A file server dies mid-push. The failure path. A server crashes after the pack is streamed but before the vote; the push still commits on the other two replicas, reads move away from the dead server, and its 800 k replicas are rebuilt elsewhere within the hour.
   1. Replica fs-0207 stops answering while a push to acme/api is between streaming and prepare. The router waits a bounded time for votes, 500 ms within a region, rather than for the slowest replica. fs-0412 and fs-1093 vote the same hash, which is a quorum.
   2. The router commits on the two healthy replicas, marks fs-0207 out of sync for this repository, and acknowledges the push. The client sees an ordinary success. The out-of-sync mark is recorded before the ack, so no reader can be routed to fs-0207 and see the old main, even if it comes back a second later. With two in-sync replicas left, the next push still reaches a quorum.
   3. Missed heartbeats declare the whole server dead, and every network with a replica on it drops to two. About 30 s of missed heartbeats, so a long pause does not trigger 20 TB of copying. Around 800 k replicas are affected; reads move to the other two replicas and writes still reach a quorum, so no repository becomes unavailable.
   4. The router queues the affected networks for repair, most urgent first, and picks a new home for each. First any network now down to one in-sync replica, which cannot take writes; then recently active networks; then the long tail. Destinations are chosen by free disk, CPU and rack, never a rack that already holds a replica of the network.
   5. Each new host copies its networks from a healthy replica with git, hundreds of copies in parallel across the fleet. 20 TB spread over ~300 destination hosts is ~70 GB each; at a throttled 100 MB/s that is about 12 minutes, and the throttle exists so repair never hurts live traffic. From the start the new replica receives every push as a non-voting member, so when the copy finishes it is at most one catch-up fetch behind.
   6. The router compares the new replica’s checksum with the healthy ones and admits it as a voting, readable replica. Equal checksums over every (ref, value) pair mean identical logical contents, whatever the pack layout. If fs-0207 comes back later, its replicas are stale copies of networks that already have three members, and they are deleted rather than trusted.
5. A pull request from a fork, merged and indexed. The background path. Pull request metadata is rows in MySQL, but every diff and merge is computed by git on the file servers, mergeability is recomputed in the background whenever either side moves, and code search catches up from the push stream.
   1. A contributor opens a pull request from a branch in their fork into the parent’s main. The fork and the parent belong to one network with one object store, so the fork’s commits are already on the parent’s file servers and nothing is copied. The web tier writes the PR row and creates refs/pull/42/head in the parent pointing at the fork’s commit, which is how anyone can fetch a pull request.
   2. A PR event goes onto the bus, and a PR worker asks a replica to build the test merge. git merge-tree --write-tree runs merge-ort entirely in memory with no working copy, and writes the merge commit as refs/pull/42/merge. GitHub has written that moving to merge-ort turned merges that took 5 s or more into ones under 200 ms, which is what makes building them speculatively affordable.
   3. The worker records the result, keyed by the base and head commits it was computed for. Storing (base_sha, head_sha) with the result means a result can never be applied to the wrong pair: if either side has moved, the page shows “checking” instead of a stale answer.
   4. A reviewer opens the diff, which the web tier computes from the merge base on the file servers. The list of changed paths comes from comparing two trees, which is cheap; each file’s diff is rendered when it scrolls into view, with size limits. Rendered diffs are cached by (base blob, head blob, options), which never goes stale.
   5. The reviewer approves and presses merge; main moves to the prepared merge commit through the same replicated ref transaction as a push. The transaction’s old value is the main the test merge was built on, so if main moved a moment ago the update fails on every replica and the PR is re-checked instead. Branch protection (approvals, required checks on the head commit) is evaluated in the prepare phase.
   6. The merge’s ref update goes onto the bus like any push: webhooks fire, and every other open PR against main becomes stale. One row records main’s new commit. Every PR’s stored result names the base it was computed against, so all 5,000 open PRs on a monorepo become stale without 5,000 writes, and workers recompute the ones people are looking at first.
   7. The code search indexer reads the push event, finds the changed blobs and indexes only the ones it has never seen. Only the default branch is indexed. A blob that is already indexed, because a fork or another repository has the same file, just gains a repository in its list. GitHub has described its Blackbird index this way, sharded by blob id, at a scale of 45 M repositories, 115 TB of content and 15.5 B documents (2023).
   8. A developer searches code; the query carries the repositories they can read and is filtered inside the index. For most users that is “public, plus a few hundred private repositories”, passed as a filter evaluated during the search, not after it. People with access to tens of thousands of repositories are filtered by organisation instead, and the final page of hits is re-checked against permissions so a just-revoked access never shows a result.

Deep dives

The theory behind it

• Replication and consistency: Leader-follower, multi-leader and leaderless replication; synchronous vs asynchronous; quorums; CAP and PACELC; consistency models from eventual to linearizable; read-your-writes and failover.
• The thundering herd problem: When many clients act at the same moment and overwhelm a system: cache stampedes, retry storms, reconnect storms after outages, synchronised cron jobs and notification opens, cold starts after deploys, and the fixes: jitter, request coalescing, early refresh, backoff, admission control and staggering.
• Merkle trees: How hash trees let systems compare and verify large datasets cheaply: structure and root hashes, finding differences between replicas, proofs of inclusion, uses in databases' anti-entropy, file sync, Git, certificate transparency, blockchains and audit logs, and design choices for building them.

Related

• Design a Distributed Cache
• Design Ticketmaster
• Design a Distributed Message Queue
• All 83 system design interview questions
• Run this question for a friend: the interviewer kit