Design YouTube

Upload, transcode, and stream video to a billion viewers: a processing pipeline on one side, a CDN problem on the other.

Difficulty: hard. Patterns: transcoding, cdn, adaptive-bitrate, pipelines, storage. Reported at Google, Netflix, Meta, Amazon, TikTok, Microsoft.

Study shows every answer; Practice hides them until you have produced your own.

Functional requirements

• Upload a video. Files up to several GB, resumable from a browser or phone, with a title, description and visibility. The video is watchable within minutes of the upload finishing.
• Watch a video. Playback starts in under two seconds anywhere in the world and adapts to the viewer's bandwidth without stalling. Seeking is instant.
• Video metadata and listing. Title, description, channel, duration, thumbnails, view count. Channel pages and a home page list videos; search is a separate system we only integrate with.
• View counting. Approximate counts on the page, accurate counts for creator analytics and payouts. Counts must resist simple replay inflation.
• Comments and likes. Modelled as a simple append and counter problem; we design the data path, not moderation.
• Visibility and takedown. Private, unlisted, public. A takedown must stop playback everywhere within minutes.
• Out of scope. Live streaming (mention how the pipeline changes), recommendations and ranking, search relevance, DRM licensing, monetisation and ads.

Non-functional requirements

• Scale (1 B DAU · 5 B views/day · 500 h uploaded/min). Reads outnumber writes by orders of magnitude and each read is megabytes per second. Bandwidth, not requests, is the cost.
• Start-up latency (< 2 s to first frame). Requires the first segments to be at the edge and the manifest tiny. This drives the CDN and segment-size design.
• Smooth playback (rebuffer ratio < 0.5 %). Adaptive bitrate must step down before the buffer empties. The hardest tradeoff: quality against stalls on variable networks.
• Upload to available (< 10 min for a 10-minute video). Transcoding is parallelised by chunk so wall-clock time does not scale with duration.
• Durability (no lost uploads). The original is the only copy the creator may have. It is written to replicated object storage before any acknowledgement.
• Availability (99.99 % for playback). Playback degrades gracefully: if metadata services are down, cached manifests still play.
• Cost (egress and storage dominate). Every design decision is checked against bytes stored per video and bytes shipped per view.

Back-of-envelope estimates

• Views per second: ~60 k avg · ~200 k peak. 5 B views/day ÷ 86 400 s ≈ ~58 k/s. Peak 3× in evening hours per region. Each view is a manifest fetch plus tens to hundreds of segment fetches from the CDN.
• Egress bandwidth: ~150 Tbit/s. Assume an average bitrate of 3 Mbit/s across quality levels and 50 M concurrent viewers at peak: 50 M × 3 Mbit/s = 150 Tbit/s. This is why the CDN is the design, not a box on it.
• Upload volume per day: ~720 k hours · ~2 PB raw. 500 h/min × 1 440 min = 720 000 h/day. Raw uploads average ~3 GB per hour of video (phone 1080p), so ~2 PB/day of originals arrive.
• Transcoded storage per day: ~1.5 PB. Each video becomes ~6 renditions (144p to 4K). Total output is roughly 0.7× the original size after modern codecs: 2 PB × 0.7 ≈ 1.5 PB/day, ~550 PB/year plus originals kept cold. Storage tiering is mandatory.
• Transcoding compute: ~60 k cores continuously. Transcoding 1 h of video into all renditions takes roughly 2 core-hours of a modern CPU (hardware encoders are faster). 720 k h/day × 2 = 1.44 M core-hours/day ÷ 24 ≈ 60 k cores busy all day, more at peak upload hours.
• Segments per view: ~150. A 10-minute video in 4-second segments is 150 segments per rendition. A viewer fetches ~150 segment files plus a manifest and, on quality switches, a few extra. At 60 k views/s that is ~9 M segment requests/s at the edge.
• View events per day: 5 B · ~1 TB. One view event of ~200 B per view: 5 B × 200 B = 1 TB/day of events, plus heartbeat events for watch time (every 30 s while playing), roughly 10× that. Fine for a streaming pipeline; not fine for a row-per-event database.

Components

• Player / uploader: The web or mobile app. Uploads in resumable chunks directly to object storage using pre-signed URLs. Plays video with an adaptive-bitrate player that reads a manifest and fetches segments from the CDN, switching quality based on measured throughput and buffer level.
• CDN (multi-tier edge cache): Serves manifests, segments and thumbnails from PoPs near viewers. Multi-tier: edge → regional shield → origin, so a popular video is fetched from origin once per region. Signed URLs enforce visibility; short TTLs on manifests make takedowns fast.
• API gateway: Authenticates, rate limits, and routes upload, metadata and engagement calls. Playback bytes never pass through it.
• Upload service: Creates the video record, issues pre-signed multipart upload URLs, tracks chunk completion for resumability, and on completion validates the file and enqueues transcoding. Never touches the bytes itself.
• Raw store (S3 / GCS · originals): Originals as uploaded, replicated across zones. Written before any acknowledgement, retained in a cold tier after transcoding so re-encoding with a better codec is possible later.
• Transcode queue (Kafka / SQS · per chunk): One job per (video, chunk, rendition). A 10-minute video becomes hundreds of small independent jobs, which is what makes wall-clock transcoding time independent of duration. Priority lanes for popular channels.
• Transcoder fleet (ffmpeg / hardware encoders): Stateless workers that pull a job, fetch the source chunk, encode one rendition, and write the segment. Also produce thumbnails, audio tracks and subtitles. Autoscaled on queue depth; spot instances are fine because jobs are small and idempotent.
• Pipeline orchestrator (DAG per video): Splits the original into chunks at keyframes, fans out jobs, tracks completion per rendition, then stitches segment lists into manifests and marks the video ready. Handles retries and partial failure; the state machine is durable.
• Segment store (object storage · CDN origin): Transcoded segments (4 s each) per rendition, plus manifests and thumbnails, laid out as immutable objects under a content-addressed path. Origin for the CDN. Tiered: hot for recent and popular, cold for the long tail.
• Metadata service: Video and channel records: title, description, status, visibility, renditions available, duration, thumbnail ids. Serves the watch page and listing pages. Emits change events on publish and takedown.
• Metadata DB (sharded MySQL / Spanner · by video id): Source of truth for videos, channels, and visibility. Sharded by video id; channel → videos is a secondary index table. Vitess-style sharded MySQL is what YouTube actually runs.
• Metadata cache (Redis · watch page objects): Watch-page metadata for the hot set of videos. A video page is read millions of times per write, so the cache hit rate is near 100 %. Invalidated on publish, edit and takedown.
• View event stream (Kafka · key=video id): View starts, heartbeats and engagement events from players, batched by the client and acknowledged by an ingest endpoint. Feeds counting, analytics and recommendations.
• View counter (stream aggregation + dedupe): Deduplicates view events per (viewer, video, window), filters obvious bots, and maintains approximate public counts (updated every few seconds) and exact daily counts for analytics. Public counts are eventually consistent by design.
• Analytics store (ClickHouse / BigQuery): Watch time, retention curves, traffic sources per video per day. Written in batches from the stream; read by creator dashboards, never by the watch page.

User flows

1. Upload a video. Bytes go straight to object storage; the services only coordinate. Resumable, durable before acknowledgement, and the pipeline starts the moment the last chunk lands.
2. Transcode into renditions. Split the original at keyframes, encode every (chunk, rendition) pair independently on a large stateless fleet, and stitch the results into manifests. Parallelism makes a two-hour film finish in minutes.
3. Watch a video. Metadata from a cache, bytes from the CDN, quality chosen by the player. The origin is touched only on cache misses; the watch page is the same for everyone so it caches perfectly.
4. A video goes viral: 5 M concurrent viewers. One video, one manifest, 150 segments per rendition, five million players. The CDN absorbs it by design; the only origin-side work is keeping metadata and counts from becoming hot spots.
5. Takedown and visibility change. Bytes are cached in a thousand places with year-long TTLs. The design makes the manifest, not the segments, the enforcement point, and signs URLs where it matters.

Deep dives

1. Parallel transcoding pipeline. Why not transcode each video as one job? How do you make a two-hour upload available in minutes?
2. Adaptive bitrate and segment size. How do you start playback in under two seconds and avoid rebuffering on a network that varies second to second?
3. Storage layout and tiering. Half a petabyte a day forever. How do you store it, and how do you keep the bill sane?
4. Counting views at scale. Why is the view count "approximate", and how do you count accurately for creator payouts without inflating on replay?
5. Metadata storage and the watch page. What holds video and channel metadata, how is it sharded, and why does the watch page not hit it?