Media uploads and processing pipelines
How photo and video uploads work at scale: direct-to-storage uploads with presigned URLs, resumable and chunked uploads, processing pipelines for thumbnails and transcoding, metadata and status tracking, deduplication, moderation hooks and serving variants through a CDN.
Reading is half of it. See this used in a real interview: walk through Design Instagram →
Uploading a photo or video looks instant in a good app, but behind it is a pipeline: get large bytes from a phone on a flaky network into storage, process them into many variants, check them, record metadata, and serve them fast worldwide. Instagram, YouTube, WhatsApp and Dropbox questions all include this pipeline, and getting the shape right (never pushing bytes through your app servers) is an easy win.
Do not proxy bytes through app servers
Streaming a 2 GB video through your API servers ties up connections and bandwidth on the most expensive tier. Instead:
- The client asks the API for an upload:
POST /uploadswith type and size. - The API checks auth and quotas, creates a pending media record, and returns a presigned URL (a time-limited, signed permission to write one object) for object storage.
- The client uploads directly to object storage (or an upload edge close to it).
- Storage emits an event when the object is complete, or the client calls
POST /uploads/{id}/complete.
App servers handle only small JSON requests. See object storage and files.
Resumable and chunked uploads
Mobile networks drop. For anything larger than a few megabytes:
- Split into chunks (5 to 10 MB), upload them independently, possibly in parallel, then assemble (multipart upload in S3-style storage).
- Resume after failure by asking which chunks already arrived, instead of starting over.
- Verify integrity with a checksum per chunk and for the whole file.
The tus protocol and cloud multipart APIs implement this. See Design Dropbox.
The processing pipeline
When the upload completes, an event starts asynchronous processing:
| Step | Photos | Videos |
|---|---|---|
| Validate | real image, allowed type, size limits | real video, duration and size limits |
| Strip metadata | remove GPS and EXIF location for privacy | same |
| Variants | thumbnails and several sizes, modern formats like WebP and AVIF | transcode to multiple resolutions and bitrates, segment for streaming |
| Analyse | moderation, hashing, labels | moderation, thumbnails from frames, captions |
| Publish | mark ready, notify followers | mark ready when the first renditions finish |
Each step runs as a job on a queue with retries and idempotency, so a crashed worker does not lose or duplicate work. Heavy video transcoding splits the file into segments processed in parallel across many workers, often on cheaper interruptible capacity. See background jobs, video streaming and cost-aware system design.
Status and user experience
Track state per media item: pending_upload, uploaded, processing, ready, failed. The client shows an optimistic local preview immediately, and the post becomes visible to others once the media is ready (or with a placeholder). Notify the client by push or polling when processing finishes. YouTube-style platforms publish low resolutions first and add higher ones as they finish.
Deduplication
Hash the content (a cryptographic hash of the file, or of each chunk). If the hash already exists, skip storing and processing it again: forwarded WhatsApp media and re-shared memes are stored once. Keep reference counts or a reference table so shared objects are deleted only when unused. See hashing and encoding and Design WhatsApp.
Safety hooks
The pipeline is where uploads get checked: perceptual hash matching against known illegal content, classifiers for nudity or violence, and malware scanning for documents. High-confidence matches block publication; uncertain ones go to review. See trust and safety.
Serving
- Store original and variants in object storage with keys like
media/{id}/{variant}. - Serve through a CDN; variants are immutable, so cache them forever and change the key when content changes.
- Pick the variant by device and network (responsive image sizes, adaptive bitrate for video).
- Private media uses short-lived signed URLs or signed cookies at the CDN.
See CDN and edge.
Storage lifecycle
Most media is viewed heavily in its first days and rarely afterwards. Move old originals and unpopular variants to colder storage tiers, and consider regenerating rarely used variants on demand instead of storing them all. Deleting a post must delete or unreference all variants and purge CDN caches. See privacy and data deletion.
In the interview
For Design Instagram or Design YouTube: presigned direct uploads with chunking and resume, an event-driven pipeline of idempotent jobs (validate, strip metadata, variants or transcode, moderate), a status machine for the media item, deduplication by hash, CDN serving of immutable variants, and lifecycle tiering.
Checklist
- Presigned URLs; bytes never pass through app servers.
- Chunked, resumable uploads with checksums.
- Event-driven, idempotent processing jobs; parallel transcoding.
- Media status machine and optimistic client previews.
- Content-hash deduplication with reference counting.
- Moderation and metadata stripping in the pipeline.
- Immutable variants on a CDN; signed URLs for private media; tiered storage.