Compress Files with Zstandard (zstd)

Zstandard is an impressive compression tool that Proxmox supports, as well as the PostgreSQL 15 (release notes) for its in-built pg_basebackup tool. There is a great video from Ville Tainio where he explains pretty well the impact of Zstd, especially by comparing it against other compression tools like gzip.

Suffice to say, wherever I was using gzip in the past, I will probably be using this instead now.

Installation

sudo apt-get install zstd

Compress A Single File (Simple)

The following is a very simple way to compress a single file

zstd --compress myfile.qcow2

This will output a version of the file with the .zst extension.

Output

Whilst compressing, you will be shown both the amount of bytes read out of the total, as well as a percentage, which is the current compression percentage, not the progress percentage. At the end, it will give you the total summary, indicating what percentage size of the original, the new file is. E.g.

disk1.qcow2          : 21.71%   (47083225372 => 10221799708 bytes, disk1.qcow2.zst)

In this case, the new file is 21.71% the size of the original. Now my backup server can store nearly 5x the number of backup images.

Compress A Single File (Advanced)

The following is a much more advanced way to compress a single file:

zstd \
  --compress \
  --threads=4 \
  --rm \
  -19 \
  myfile.qcow2

• -rm will result in deleting the original source file after having created the compressed version of the file.
• -19 tells it to use compression level 19 (highest without requiring another perameter to use more memory).
• --threads=4 tells it to use 4 threads, instead of just 1.

Decompress A Single File

The following is a very simple way to decompress a single file

zstd --decompress myFile.qcow2.zst

Zstd does not support using multiple threads for decompression.

One can also use --uncompress, in case you forget --decompress

zstd --uncompress myFile.qcow2.zst

Compress Folder

export ZSTD_CLEVEL=10
export ZSTD_NBTHREADS=4
tar --create --zstd --file myFolder.tar.zst myFolder

The compression level variable can be a number between 1 and 19. The default is 3. Higher = more compressed, but slower.

If you want an alternative way to perform the same action, you can use:

# Set the compression level. An integer between 1 and 19 with 19 being the highest
# if not specifying --ultra for increased memory usage. Higher =  more compressed.
COMPRESSION_LEVEL=19

# Specify the number of threads to use. This should probably be equal to the number
# of threads your CPU supports.
NUM_THREADS=4

tar \
  --use-compress-program "zstd --threads=$NUM_THREADS -$COMPRESSION_LEVEL"\
  --create \
  --file myFolder.tar.zst myFolder

Decompress

One can then extract the archive by running:

export ZSTD_NBTHREADS=4
tar --extract --zstd --file myFolder.tar.zst

Zstandard decompression is so fast that I couldn't really tell if it was really using the threads.

References

• Github - facebook/zstd - Make number of threads configurable via Environment Variable
• Hacker news Post #21959517
• Linux Hint - How to compress files with zstd from the command line or bash script