Choosing between LZ4, zstd and gzip on fast and slow links: an interactive compression comparison
Compression trades CPU time for fewer bytes. Fewer bytes mean less time on the network, less storage and smaller bills, but only if compressing and decompressing keep up. Low-level zstd shrinks data almost as much as gzip while compressing more than ten times faster, LZ4 gives up some ratio for even more speed, and maximum-ratio settings are so slow they only make sense when data is compressed once and read many times.
The playground moves 1 GB of logs over links from 10 Mbps to 10 Gbps with five codecs on one core. Compression, transfer and decompression are streamed, so the slowest stage sets the pace.
Moving 1 GB of logs over a 100 Mbps link, compressed for this transfer, on one core. zstd level 1 is fastest at 28 s, against 80 s uncompressed, sending 345 MB instead of 1,000. gzip level 6 and zstd level 19 are limited by compression speed, not the link: a better ratio is worthless if the compressor cannot keep up.
What decides the winner
- A slow link rewards ratio. At 100 Mbps, zstd level 1 moves the gigabyte in 28 s against 80 s uncompressed. gzip has a better ratio but compresses at about 30 MB/s on one core, so it becomes the bottleneck at 33 s, and zstd 19 takes 250 s.
- A fast link punishes CPU. At 10 Gbps the raw transfer takes 0.8 s and no codec on one core compresses that fast: LZ4 takes 1.4 s. Inside a datacenter, compress with several threads, compress less, or skip it.
- Compress once, read many. A file built once and downloaded a thousand times pays compression once, so the slowest, tightest setting wins every read: zstd 19 serves it in 2.2 s at 1 Gbps against 8 s raw. This is why static assets are precompressed with brotli or zstd at maximum level.
- Storage and egress scale with the ratio. A terabyte of raw logs costs $23 a month in object storage uncompressed and $6.39 at zstd 19’s ratio, and egress falls the same way.
- Data matters most. Text, JSON and logs compress 3 to 10 times; images, video and encrypted data are already dense and do not compress at all. Columnar formats compress far better than rows because similar values sit together.
Where the setting lives
# Kafka producer: compress batches on the way in; brokers store them compressed
compression.type=zstd # or lz4 when producers are CPU-bound
linger.ms=20 # bigger batches compress better
batch.size=262144
# HTTP: precompress static files once at build time, serve the best one the client accepts
# app.3f9a1c.js.br (brotli, max level, built once)
# app.3f9a1c.js.gz (gzip fallback)
Content-Encoding: br
# command line: zstd on all cores for a big archive
zstd -19 -T0 logs-2026-10.tar -o logs-2026-10.tar.zstIn a system design interview
Mention compression when bytes dominate the cost: “Producers compress batches with zstd before sending to Kafka, which cuts network and broker storage about threefold at a little CPU. Cold logs go to object storage as zstd-compressed Parquet. Static assets are precompressed at build time with brotli. Inside the datacenter between services we use LZ4 or nothing, because the network is not the bottleneck.” Expect the follow-ups: what is compressible, CPU cost at peak, and where compression breaks random access, such as seeking inside a large compressed file.
Read the guide to compression or cost optimisation, see row vs column storage and HTTP caching, or watch a strong candidate design a metrics system.
Questions people ask
Which is better, zstd or gzip?
For most new systems zstd: at low levels it compresses about as well as gzip while running many times faster, and its decompression is faster too. gzip remains the most widely supported format, for example as an HTTP Content-Encoding every client accepts.
When should I use LZ4?
When CPU time matters more than size: fast networks, in-memory or on-disk data read constantly, and hot paths inside a datacenter. LZ4 gives a lower ratio than zstd or gzip but compresses and especially decompresses extremely fast.
Does compression make data transfer faster?
Only when the network is the bottleneck. If the link moves bytes faster than one core can compress them, as on 10 Gbps links inside a datacenter, compression slows the transfer down unless it uses several cores.
What is a compression ratio?
The original size divided by the compressed size. Text, JSON and logs typically compress 3 to 10 times depending on the codec and data, while images, video and encrypted data are already dense and barely compress.
Why precompress static files?
Files built once and served many times can use the slowest, highest-ratio settings, such as brotli or zstd at maximum level, because compression happens once at build time while every download benefits from the smaller size.
Should Kafka messages be compressed?
Usually yes: producers compress whole batches, brokers store and replicate them compressed, and consumers decompress, so one CPU cost saves network, disk and replication. zstd or lz4 are common choices, and larger batches compress better.
Why do columnar formats compress so well?
Values from one column are stored together, so neighbouring bytes are similar: repeated categories, sorted timestamps, small numbers. Encodings such as dictionary, run-length and delta make them smaller still before a general-purpose codec runs.