Qvm-backup compression filter execution time and output size comparison

General Discussion
1

Hi,

I was curious to know how to improve the default backup tool. I saw gzip was CPU bound and thought at proposing pigz as a replacement, because it’s in fedora repository and doesn’t change the output format, so it’s still easy to decompress for unattended backup restore.

Then, I thought it would be nice to compare with xz because it’s already there, and I remembered reading about someone asking to use zstd, but it was dismissed because zstd isn’t always available in you need to restore your backups in emergency, so I added to the list for more fun.

qvm-backup
qvm-backup892×561 23.5 KB

My conclusion is it would be interesting to use pigz instead of gzip as a default compression filter, using 2 or 3 cores in its command line. This would only require Qubes OS to add this small package in the default installation. There is no drawback, and the output remains a gzip file.

Trivia: qvm-backup --compress-filter=params has an issue, params can’t have parameters despite tar --use-compress-program=params being compatible with parameters. I had to use a shell script with the command in it, like xz -T 5 --fast and used that script in --compress-filter=/my/script.

3 Likes
2

Others have made the same request Qubes 4.0 Backup VMs slooow (gzip) - #21 by luja

With 2 cores, there wasn’t much difference in performance, but maybe it’s more relevant now when people start getting system with much higher core counts.

3

On 2 cores, you either have the compression tool or the encryption tool that will fill your CPU usage, so there is just no room to make it faster, except using zstd that has been using almost nothing during my tests. I didn’t measure CPU usage because it would have required more work I didn’t want to do, but zstd was using 15% of one core at best.

I’ve been running the tests on a desktop Ryzen 5600X 6C/12T (smt=on).

4
1 Like
5

I usually use bzip2 when I want to save space.