Storage tiering and lifecycle rules for hot, warm and cold data: an interactive cost simulator

Object stores offer several storage classes. The cheaper ones store a gigabyte for a fraction of the price but charge to read it back, and the cheapest make a read wait hours. Lifecycle rules move objects to cheaper classes as they age. Whether that saves money depends entirely on how often old data is still read.

The playground stores 30 TB of new images a month for three years, about a petabyte in the end, under four policies, with approximate S3 prices. Switch between user photos, which go cold within weeks, and product images, which are read steadily forever.

When tiering pays

• Cold data is where the money is. For user photos, keeping everything in Standard costs about $460K over three years. Moving photos to Infrequent Access after 30 days and Glacier Instant Retrieval after 90 cuts that to about $142K, and every read still returns in milliseconds.
• Deep archive is for data nobody waits for. Deep Archive takes it to about $112K, but about a quarter of all views would then wait up to 12 hours for a restore. It suits backups, compliance archives and raw logs, not anything a user can click on.
• Retrieval fees punish warm data. Product images read 1.5 times a month at any age cost more in every cheaper class: $548K in Infrequent Access and $917K in Glacier Instant Retrieval against $460K in Standard, because each read pays a per-gigabyte fee that the storage saving cannot cover.
• Small objects and short lives cost extra. Transitions are charged per object, and the cheaper classes bill a minimum size and a minimum storage duration, so millions of tiny files, or files deleted soon after moving, can cost more than they save.

A lifecycle configuration

{
  "Rules": [{
    "ID": "photos-cool-down",
    "Filter": { "Prefix": "photos/" },
    "Status": "Enabled",
    "Transitions": [
      { "Days": 30, "StorageClass": "STANDARD_IA" },
      { "Days": 90, "StorageClass": "GLACIER_IR" }
    ]
  }, {
    "ID": "backups-to-archive",
    "Filter": { "Prefix": "backups/" },
    "Status": "Enabled",
    "Transitions": [{ "Days": 7, "StorageClass": "DEEP_ARCHIVE" }],
    "Expiration": { "Days": 2555 }
  }]
}

When access patterns are unknown, an automatic class such as S3 Intelligent-Tiering watches each object and moves it between tiers for a small monitoring fee per object, which works well for large objects with unpredictable access.

In a system design interview

When the estimate reaches petabytes, say how the bill is controlled: “Photos are read heavily in their first weeks, so after 30 days a lifecycle rule moves them to infrequent access and after 90 days to an instant-retrieval archive class; reads stay fast and the storage bill drops by about three quarters. Original uploads we only need for reprocessing go to deep archive. Thumbnails stay in the standard class because they are read constantly.” Expect the follow-ups: what happens when an old photo goes viral, how deletes interact with minimum durations, and whether a CDN in front changes the read pattern the store sees.

Read the guide to object storage and files or cost optimisation, see erasure coding and CDN caching, or watch a strong candidate design Instagram.

Questions people ask