Multipart and resumable uploads: an interactive simulator over a flaky connection

Uploading a large file in one request works until the connection drops, and then everything sent so far is lost. Over a mobile network or a long transfer, drops are a certainty, not a risk. Resumable uploads let the client continue from the last byte the server acknowledged, and multipart uploads split the file into parts that are sent in parallel and retried individually, which is how S3, Google Cloud Storage and most media apps move big files.

The playground uploads a file three ways over connections that drop at random. Change the file size and how often the connection fails.

Why one PUT fails at scale

At 10 MB a second, a 1 GB file takes 100 seconds in one piece. If the connection drops about once a minute, an unbroken 100 seconds is unlikely, and every drop starts the upload over: 8 restarts and 4 GB sent for nothing before it finally succeeds after nearly nine minutes. A 5 GB file needs over eight unbroken minutes, which essentially never happens: after an hour and 35 GB of traffic, the server still has nothing. The expected time grows exponentially with the file’s transfer time divided by the time between failures.

Resumable and multipart uploads

• Resumable uploads (the tus protocol, Google’s resumable upload API) keep one session; after a drop the client asks how much the server has and continues from there. The 1 GB file takes 102 seconds and wastes 3 MB.
• Multipart uploads (S3, GCS XML API, Azure blocks) send numbered parts, here 16 MB each, four at a time, and a final call stitches them together. A drop costs only the part in flight, and parallel connections use more bandwidth: 27 seconds for 1 GB, 131 seconds for 5 GB.
• Clean up. Unfinished multipart uploads keep their parts, and their storage cost, until aborted; set a lifecycle rule that removes them after a few days.
• Upload straight to storage. The app server issues pre-signed URLs per part and the client sends bytes directly to object storage, so large files never pass through the app servers.

A parallel multipart upload to S3

import os
import boto3
from concurrent.futures import ThreadPoolExecutor

s3 = boto3.client("s3")
PART = 16 * 1024 * 1024                                   # every part but the last >= 5 MB

def upload(path, bucket, key):
    upload_id = s3.create_multipart_upload(Bucket=bucket, Key=key)["UploadId"]
    size = os.path.getsize(path)
    def send(n):
        with open(path, "rb") as f:
            f.seek(n * PART)
            for attempt in range(5):                      # a failed part is retried alone
                try:
                    etag = s3.upload_part(Bucket=bucket, Key=key, UploadId=upload_id,
                                          PartNumber=n + 1, Body=f.read(PART))["ETag"]
                    return {"PartNumber": n + 1, "ETag": etag}
                except Exception:
                    f.seek(n * PART)
            raise RuntimeError(f"part {n + 1} kept failing")
    with ThreadPoolExecutor(4) as pool:                   # 4 parts in flight
        parts = list(pool.map(send, range((size + PART - 1) // PART)))
    s3.complete_multipart_upload(Bucket=bucket, Key=key, UploadId=upload_id,
                                 MultipartUpload={"Parts": parts})

In a system design interview

For video, photo or file-sync designs, say how bytes get in: “The client asks the upload service for a multipart upload and pre-signed URLs for each 16 MB part, uploads four parts at a time directly to object storage, retries failed parts, and calls complete when all ETags are in. If the app is killed, it resumes by listing the parts already uploaded. Completion triggers an event that starts transcoding.” Expect the follow-ups: verifying integrity with checksums, deduplicating identical uploads, and limiting abuse of pre-signed URLs.

Read the guide to media uploads and processing or object storage, see content-defined chunking for uploading only what changed, or watch a strong candidate design a video streaming service.

Questions people ask