Parallel video transcoding with chunks, worker pools and backup tasks: an interactive simulator
Encoding a long video into every resolution a streaming service needs takes hours on one machine. Splitting it into chunks at keyframes lets hundreds of workers encode in parallel, the way MapReduce splits a big job. The chunk size decides how well that works: too large and there is not enough parallelism and one slow machine holds everything up, too small and per-chunk overhead eats the gain. Backup tasks deal with the slow machines.
The playground transcodes a two-hour film that needs 2 s of worker time per second of video, four hours on one worker. Each chunk also costs 5 s to start, fetch and upload, and one task in twenty lands on a machine four times slower.
Split into 12 chunks of 10 min across 50 workers, the film is ready in 81 min instead of 4 h, including 30 s to split and 20 s to stitch. There are fewer chunks than workers, so 38 workers have nothing to do and the job cannot finish faster than one chunk. 2 chunks land on a slow machine and take 4 times as long; the film is not ready until the last one is.
What the chunk size does
- The whole film on one worker takes 4 hours, however many workers are idle.
- Ten-minute chunks give only 12 tasks, so 50 workers cannot help beyond 12, and one chunk on a slow machine takes 80 minutes alone: the film is ready after 81 minutes instead of 21. Backup tasks bring it back to 41.
- One-minute chunks on 200 workers finish in 9 minutes, still waiting on slow machines; with backups, 5.
- Ten-second chunks with backup tasks are ready in under 3 minutes: 720 tasks keep 200 workers busy, a slow chunk costs little, and 71% of worker time is still encoding.
- Two-second chunks go too far: 3,600 tasks each paying 5 s of overhead, so most worker time is spent starting, fetching and uploading, and the job is slower than with ten-second chunks.
The pipeline
upload.mp4 ──► split at keyframes ──► chunk 0001 … chunk 0720 (10 s each, in object storage)
│
queue of encode tasks, one per chunk × ladder
│
workers pull a task, encode 1080p/720p/480p/…, upload the segments
│
all chunks done? ──► stitch / write the manifest ──► publish
# a task is idempotent: its output path is derived from (video, chunk, rendition),
# so a retried or duplicated task overwrites the same file instead of adding oneChunks must start on keyframes, which streaming already requires because every segment must be playable on its own; encoders are told to place keyframes at fixed intervals, every 2 to 4 seconds. Popular titles can be encoded again later with slower, better settings, since compression time is paid once and the savings on every view.
In a system design interview
For a video platform, say how an upload becomes streamable quickly: “We split the upload into chunks of a few seconds at keyframes and fan them out to a worker pool through a queue, one task per chunk and rendition. Tasks are idempotent, so retries and backup copies of slow tasks are safe. When all chunks finish, we write the manifest and publish. A two-hour film is streamable in minutes instead of hours.” Expect the follow-ups: what happens when a worker dies, how the low resolutions can be published first, and how to prioritise a popular creator’s upload.
Read the guide to media uploads and processing or MapReduce and Spark, see adaptive bitrate streaming and tail latency, or watch a strong candidate design a video platform.
Questions people ask
How do video platforms transcode videos so fast?
They split each upload into short chunks at keyframes and encode the chunks in parallel on many workers, one task per chunk and output rendition. A film that would take hours on one machine is ready in minutes, then the chunks are stitched or referenced from a manifest.
What chunk size should a transcoding pipeline use?
Large enough that per-chunk overhead, such as starting a task and fetching and uploading data, is small compared with encoding time, and small enough that there are many more chunks than workers and one slow chunk does not delay the whole video. A few seconds to a minute is typical.
What are stragglers?
Tasks that run much slower than others because of a slow or overloaded machine, a noisy neighbour or a failing disk. In a parallel job the result is only ready when the last task finishes, so a few stragglers can dominate the total time.
What are backup tasks or speculative execution?
When most tasks are done, idle workers start duplicate copies of the ones still running, and whichever copy finishes first is used. MapReduce introduced it to cut the tail caused by stragglers; it costs a little duplicated work.
Why must video chunks start at keyframes?
A keyframe can be decoded on its own, while other frames depend on earlier ones. Splitting at keyframes lets each chunk be encoded and played independently, which adaptive streaming needs anyway for its segments.
How do you make transcoding tasks safe to retry?
Make each task idempotent: derive its output location from the video, chunk and rendition, so a retried or duplicated task overwrites the same object. Track completion per task so the final stitch step only runs when every chunk is present.
Why encode popular videos again later?
Slower encoder settings or newer codecs produce smaller files at the same quality. The extra compute is paid once, while the bandwidth savings repeat on every view, so platforms re-encode their most watched titles.