Compress Files with Zstandard (zstd)
Zstandard is an impressive compression tool that Proxmox supports, as well as the PostgreSQL 15 (release notes) for its in-built pg_basebackup tool. There is a great video from Ville Tainio where he explains pretty well the impact of Zstd, especially by comparing it against other compression tools like gzip.
Suffice to say, wherever I was using gzip in the past, I will probably be using this instead now.
sudo apt-get install zstd
Compress A Single File (Simple)
The following is a very simple way to compress a single file
zstd --compress myfile.qcow2
This will output a version of the file with the
Whilst compressing, you will be shown both the amount of bytes read out of the total, as well as a percentage, which is the current compression percentage, not the progress percentage. At the end, it will give you the total summary, indicating what percentage size of the original, the new file is. E.g.
disk1.qcow2 : 21.71% (47083225372 => 10221799708 bytes, disk1.qcow2.zst)
In this case, the new file is 21.71% the size of the original. Now my backup server can store nearly 5x the number of backup images.
Compress A Single File (Advanced)
The following is a much more advanced way to compress a single file:
zstd \ --compress \ --threads=4 \ --rm \ -19 \ myfile.qcow2
-rmwill result in deleting the original source file after having created the compressed version of the file.
-19tells it to use compression level 19 (highest without requiring another perameter to use more memory).
--threads=4tells it to use 4 threads, instead of just 1.
Decompress A Single File
The following is a very simple way to decompress a single file
zstd --decompress myFile.qcow2.zst
One can also use
--uncompress, in case you forget
zstd --uncompress myFile.qcow2.zst
export ZSTD_CLEVEL=10 export ZSTD_NBTHREADS=4 tar --create --zstd --file myFolder.tar.zst myFolder
If you want an alternative way to perform the same action, you can use:
# Set the compression level. An integer between 1 and 19 with 19 being the highest # if not specifying --ultra for increased memory usage. Higher = more compressed. COMPRESSION_LEVEL=19 # Specify the number of threads to use. This should probably be equal to the number # of threads your CPU supports. NUM_THREADS=4 tar \ --use-compress-program "zstd --threads=$NUM_THREADS -$COMPRESSION_LEVEL"\ --create \ --file myFolder.tar.zst myFolder
One can then extract the archive by running:
export ZSTD_NBTHREADS=4 tar --extract --zstd --file myFolder.tar.zst
- Github - facebook/zstd - Make number of threads configurable via Environment Variable
- Hacker news Post #21959517
- Linux Hint - How to compress files with zstd from the command line or bash script
First published: 16th October 2022