Compression — gzip, zstd, xz and bzip2

tar bundles many files into one; it does not compress anything. Compression is a separate step done by a separate tool, and there are four of them with meaningfully different trade-offs. Picking the right one can turn a twenty-minute backup into a two-minute one.

The four, compared

They all share one interface

gzip file.log            # produces file.log.gz AND DELETES file.log
gzip -k file.log         # keep the original
gunzip file.log.gz       # or gzip -d
gzip -c file.log > out.gz  # write to stdout, original untouched
gzip -9 file.log         # maximum effort
zstd file.log            # zstd keeps the original by default
zstd --rm file.log       # ...unless you ask it not to

gzip file.log            # produces file.log.gz AND DELETES file.log
gzip -k file.log         # keep the original
gunzip file.log.gz       # or gzip -d
gzip -c file.log > out.gz  # write to stdout, original untouched
gzip -9 file.log         # maximum effort
zstd file.log            # zstd keeps the original by default
zstd --rm file.log       # ...unless you ask it not to

gzip, xz and bzip2 replace the file. The original is gone once compression succeeds, which is fine until you do it to something you needed in place. -k keeps it. zstd is the odd one out and keeps the original unless told otherwise — so the habit you build with one is wrong for the other.

Levels, and where they stop being worth it

gzip -1 file      # fastest      gzip -9 file        # smallest (barely)
zstd -3 file      # the default  zstd -19 file       # much smaller, much slower
zstd --ultra -22 file            # the absolute maximum
xz -6 file        # the default  xz -9 file          # rarely worth it

gzip -1 file      # fastest      gzip -9 file        # smallest (barely)
zstd -3 file      # the default  zstd -19 file       # much smaller, much slower
zstd --ultra -22 file            # the absolute maximum
xz -6 file        # the default  xz -9 file          # rarely worth it

For gzip, the difference between -6 and -9 is usually a couple of percent for noticeably more time — not a good trade. For zstd the range is genuinely wide and useful: -19 is a different tool from -3, and because zstd’s decompression speed barely changes with level, paying once for a high level costs the reader nothing.

One caution on xz -9: the high levels need a large dictionary, and that memory is required to decompress as well as to compress. An archive made with xz -9 can fail to open on a small machine. xz -6 is the sensible default and is what most distributions use.

Use all your cores

zstd -T0 -10 big.tar          # T0 = every core
xz -T0 big.tar
pigz big.tar                  # parallel gzip, drop-in replacement
pigz -p 4 -9 big.tar
pbzip2 big.tar                # parallel bzip2

zstd -T0 -10 big.tar          # T0 = every core
xz -T0 big.tar
pigz big.tar                  # parallel gzip, drop-in replacement
pigz -p 4 -9 big.tar
pbzip2 big.tar                # parallel bzip2

Plain gzip and bzip2 are single-threaded and always will be. On an eight-core machine, swapping gzip for pigz is close to an eight-fold speed-up for a one-word change, and the output is an ordinary .gz file that anything can read.

With tar

tar caf backup.tar.zst /srv/myapp        # -a picks zstd from the name
tar czf backup.tar.gz /srv/myapp
tar xf backup.tar.anything               # extraction auto-detects; no flag needed

# a custom compressor, with your own flags
tar -I 'zstd -T0 -10' -cf backup.tar.zst /srv/myapp
tar -I pigz -cf backup.tar.gz /srv/myapp

tar caf backup.tar.zst /srv/myapp        # -a picks zstd from the name
tar czf backup.tar.gz /srv/myapp
tar xf backup.tar.anything               # extraction auto-detects; no flag needed

# a custom compressor, with your own flags
tar -I 'zstd -T0 -10' -cf backup.tar.zst /srv/myapp
tar -I pigz -cf backup.tar.gz /srv/myapp

Two things worth knowing. Extraction needs no compression flag at all — tar xf detects the format, which makes the flag most people memorise unnecessary. And -I is how you get parallelism or a specific level, since tar’s own flags always use the default settings.

--zstd needs a reasonably recent GNU tar. On an older system, tar -I zstd works where --zstd does not.

Reading without unpacking

zcat access.log.gz | tail -50
zgrep -c 'ERROR' /var/log/*.gz          # grep straight through compression
zless huge.log.gz                       # page it
xzcat file.xz | head
zstdcat file.zst | wc -l
bzcat file.bz2 | less
zdiff old.log.gz new.log.gz

zcat access.log.gz | tail -50
zgrep -c 'ERROR' /var/log/*.gz          # grep straight through compression
zless huge.log.gz                       # page it
xzcat file.xz | head
zstdcat file.zst | wc -l
bzcat file.bz2 | less
zdiff old.log.gz new.log.gz

zgrep across rotated logs is the one to remember. Searching yesterday’s .gz files directly saves decompressing gigabytes onto a disk that may be the reason you are looking in the first place — see Reading Logs.

On most systems less handles compressed files by itself, via lesspipe. If less file.gz shows binary rubbish, the helper is not installed.

Which one, for what

Gotchas

• Compressing compressed data wastes time and can grow the file. JPEGs, PNGs, MP4s, .zip, .deb, .rpm and container images are already compressed. A tar of a photo library is a tar, not a smaller tar.
• Encrypt after compressing, never before. Encrypted data is indistinguishable from random and will not compress at all.
• Compression is not integrity. These formats carry a checksum that catches corruption on read, but nothing tells you the file was corrupted before compression. Keep a sha256sum alongside anything that matters.
• A truncated .gz is partly recoverable; a truncated .xz often is not. If a stream may be interrupted, that is an argument for gzip or zstd.
• zip is a different thing. It archives and compresses together, per file, and is what you send to someone on Windows. It is not the Unix idiom, but it is the right answer for that one case.

Measuring, rather than guessing

for c in "gzip -6" "zstd -3" "zstd -19" "xz -6"; do
  printf '%-10s ' "$c"
  /usr/bin/time -f '%es' $c -c big.log 2>&1 >/dev/null | tr '\n' ' '
  $c -c big.log 2>/dev/null | wc -c
done

for c in "gzip -6" "zstd -3" "zstd -19" "xz -6"; do
  printf '%-10s ' "$c"
  /usr/bin/time -f '%es' $c -c big.log 2>&1 >/dev/null | tr '\n' ' '
  $c -c big.log 2>/dev/null | wc -c
done

Two minutes on a representative file beats any table, including the one above — ratios vary enormously with the data. Log files and SQL dumps compress spectacularly; anything already binary barely moves.

Quick reference

zstd -T0 -10 file            # compress, all cores, keeps the original
zstd -d file.zst             # decompress
gzip -k file                 # compress, KEEP the original
gunzip file.gz
tar caf out.tar.zst dir/     # tar picks the compressor from the name
tar xf out.tar.zst           # extraction needs no flag
tar -I 'zstd -T0 -19' -cf out.tar.zst dir/
zgrep ERROR /var/log/*.gz
zcat file.gz | head

zstd -T0 -10 file            # compress, all cores, keeps the original
zstd -d file.zst             # decompress
gzip -k file                 # compress, KEEP the original
gunzip file.gz
tar caf out.tar.zst dir/     # tar picks the compressor from the name
tar xf out.tar.zst           # extraction needs no flag
tar -I 'zstd -T0 -19' -cf out.tar.zst dir/
zgrep ERROR /var/log/*.gz
zcat file.gz | head

Related

• tar — the archiver these plug into
• dd — piping a disk image through a compressor
• Disk Space — for when the answer is “compress the logs”
• restic — which handles compression and deduplication for you