Kafka log compaction vs time-based retention for changelog topics: an interactive simulator

A log that only grows is a complete history but an ever larger one. Time-based retention deletes old messages, which is right for events but wrong for state: a key that has not changed recently vanishes. Log compaction keeps the newest message for every key instead, so the topic stays about the size of the current state and a new consumer can rebuild that state by reading it from the start.

The playground follows a topic of user profiles for 60 days: every user written once on day 0, then 2,000 updates a day, mostly to a few active users, and a few account deletions. Pick a day and see what a brand new consumer could rebuild from each topic.

What each policy keeps

• Keeping everything works for rebuilding but grows without bound: by day 60 the topic holds 119,619 messages for 820 users, and every new consumer must read all of them.
• Deleting after 7 days keeps the size flat, about 14,000 messages, but it drops users who have not changed for a week. From day 7 on, a new consumer silently misses between 58 and 106 of them, depending on the day. Time retention suits events; it is the wrong tool for state.
• Compaction keeps one message per live user plus the not yet compacted active segment: about 2,800 to 3,000 messages, and every profile rebuilds. Deleting a user takes a tombstone, a message with the key and no value, which compaction keeps for a while so consumers see the delete, then removes along with the user.

Configuring a compacted topic

# a changelog topic: the newest value per key, forever
kafka-topics --create --topic user-profiles \
  --config cleanup.policy=compact \
  --config min.cleanable.dirty.ratio=0.5 \   # clean once half the log is superseded
  --config segment.ms=86400000 \             # roll the active segment daily; it is never compacted
  --config delete.retention.ms=86400000       # keep tombstones a day after cleaning

# producing: the key is what compaction keeps one of
key=user:812  value={"name":"Ada","plan":"pro"}
key=user:812  value={"name":"Ada","plan":"team"}   # supersedes the line above
key=user:977  value=null                           # tombstone: delete user 977

Compaction runs in the background and never touches the active segment, so a compacted topic still holds some superseded values; consumers must apply messages in order and let later ones win. A consumer that is offline for longer than delete.retention.ms can miss a tombstone and keep a deleted key, so that setting should exceed the longest expected outage.

In a system design interview

Compacted topics come up wherever a stream carries state: “Each service publishes its entity changes to a compacted topic keyed by entity id. A new service, or a rebuilt cache or search index, bootstraps by reading the topic from the beginning and then keeps consuming. Event history we need for audit goes to a separate topic with time-based retention and to cold storage.” Kafka itself stores consumer offsets this way, and Kafka Streams backs its state stores with compacted changelog topics. Expect the follow-ups: how deletes work, how long a rebuild takes, and what the topic looks like before compaction catches up.

Read the guide to Kafka internals or change data capture, see Kafka partitions and consumer groups and event sourcing, or watch a strong candidate design a message queue.

Questions people ask