Object storage and large files

Blob storage, pre-signed uploads, chunking and resumable transfer, multipart, erasure coding, storage tiers and lifecycle, plus how media pipelines are structured.

Every design with a photo, a video, a backup or an attachment has an object store in it. The interview points are always the same: bytes never flow through your application servers, uploads are resumable, durability comes from erasure coding rather than copies, and cold data costs a tenth of hot data.

Why a separate store

Databases are built for small rows read by index; object stores are built for large immutable blobs read by key. Put a 4 MB image in Postgres and you have burned your buffer cache, your replication bandwidth and your backup window on data that never needed a transaction.

The rule: metadata in the database, bytes in the object store, the database row holds the key.

Uploads: keep bytes out of your servers

The pattern to name in any upload design:

1. The client asks your API for permission to upload: POST /uploads with the file name, size and content type.
2. The API checks quota and content type, creates a row in uploads with state pending, and returns a pre-signed URL scoped to one key, one method, a size limit and a short expiry (minutes).
3. The client PUTs the bytes straight to the object store.
4. The store fires an event (or the client calls back) and your service flips the row to ready and kicks off processing.

Why it matters: your application servers never carry the payload, so a 1 GB upload does not occupy a request thread or your egress budget. Say that sentence — it is the point of the design.

For large files, use multipart upload: the client splits the file into parts (5–100 MB), uploads parts in parallel, retries only failed parts, and completes the upload with a list of part numbers and ETags. That is what makes an upload resumable over a flaky mobile connection.

Chunking and deduplication

File-sync and backup systems (Dropbox, Drive) chunk differently: split the file into content-defined chunks of a few megabytes, hash each chunk, and store chunks keyed by hash.

• A changed byte in the middle rewrites one chunk, not the file.
• Identical chunks across users are stored once, which is where the storage saving in a backup product comes from.
• The file becomes a manifest: an ordered list of chunk hashes, which is what syncs between devices.

Content-defined boundaries (a rolling hash such as Rabin fingerprinting) beat fixed-size blocks because inserting a byte at the start does not shift every subsequent boundary.

Durability: replication versus erasure coding

Three full copies gives 3× the storage for durability. Erasure coding splits an object into k data fragments plus m parity fragments and can rebuild it from any k: a common 10 + 4 scheme survives any four losses at 1.4× storage instead of 3×.

The tradeoff: reads need k fragments from k machines, so latency and CPU are higher and small objects suffer. Real systems store small and hot objects replicated, and large or cold objects erasure coded. Quoting "eleven nines of durability" is fine, but say what produces it: fragments spread across failure domains plus continuous background repair.

Storage classes and lifecycle

A lifecycle policy moves objects between them by age. In a video or photo design, saying "originals go to archive after 30 days, transcoded renditions stay hot, and we delete intermediates after the pipeline finishes" is a cheap and concrete win — the video question is dominated by storage cost, not compute.

Serving reads

Public and immutable objects go through a CDN with a long Cache-Control and a content-hash in the key, so the URL is the version. Private objects use short-lived signed URLs, which the CDN can still cache if you sign on the edge or key the cache on the path rather than the query string. Never serve user content from your main domain if you can avoid it: a hostile SVG or HTML file becomes an XSS on that origin.

Media pipelines

A transcoding or processing pipeline is almost always: upload → event → queue → worker fleet → derived objects → metadata update → notification.

• Split long media into segments and process segments in parallel; a 2-hour video transcodes in minutes on a big enough fleet.
• Make every step idempotent and keyed by (object key, rendition), so a retried worker overwrites the same output rather than producing a duplicate.
• Keep the job state in a database row per rendition so the UI can show progress and a partial failure retries only what failed.
• Use spot or preemptible capacity for the fleet; the work is retryable by construction.

Numbers worth carrying

• Photo upload: 2–5 MB. Phone video: ~10 MB per minute at 1080p; a source master can be 100× that.
• 1 M photos/day at 3 MB ≈ 3 TB/day ≈ 1.1 PB/year before renditions.
• Renditions typically add 30–70% on top of the original.
• Egress, not storage, is usually the bill: 1 PB served per month is a large number at any provider, which is why CDN hit rate is a cost lever.

Common mistakes

• Proxying uploads and downloads through application servers.
• Storing blobs in the database "for transactional consistency", then needing a cleanup job anyway.
• No orphan cleanup: rows without objects, or objects without rows, after a failed upload. A reconciliation job that sweeps pending uploads older than an hour is the answer.
• Signed URLs with hour-long expiries that end up shared publicly.
• Forgetting that deletes must remove every rendition, every CDN copy and every backup with a retention policy — and that "delete" in a compliance context means a documented process, not one DELETE statement.

Checklist

• Metadata in the database, bytes in the object store, key in the row.
• Pre-signed, size- and type-limited uploads; multipart for large files.
• Durability scheme and where fragments or replicas live.
• Lifecycle policy and the storage cost per year.
• CDN strategy for reads and how private content is signed.
• Orphan cleanup and delete semantics.