Qvm-backup compression filter execution time and output size comparison

OK. In this case, I believe having it shipped as the default (or recognized) compression filter might be a little bit tricky. I am not sure if Qubes OS core team would approve any software in copr due to security concerns. But there is still hope. It might be possible to contact Fedora packaging team (e.g. via their Matrix channel) and query about the status of that package.

Having said all of the above, I believe that the chances of having it in Qubes OS r4.3 is slim. As it should take some time for it to be transferred from copr to testing and stable repositories.

1 Like

I’ll try to reach out to them.

In any case: Will the interface be configurable? (Fierce wink …) If not, pigz will suffice in the meantime.

By default, the recognized compression filters are hard-coded here. And backup GUI will show those filters which are installed. But it is possible to use the CLI with any filter. It is also possible to trick the Backup GUI with the bash executable at ~/bin (for example ~/bin/xz could point to /usr/bin/plzip.

1 Like

Fair enough. So, one could easily work around things. Thank you, @alimirjamali!

1 Like

Is this advantageous? If so, “pigz” sounds good: it brings in the good old gzip utility and with multiple threads execution (at least for compression, which is where most of the user wait times exist anyways (I backup more times than I do restore))

I did a real-life comparison during my weekly backups, comparing some compression filters.

Conditions:

  • HP Elitebook 820 G1 laptop
  • Intel i5-4300u, 2 cores / 4 threads (SMT enabled in Xen config)
  • 16GB DDR3 RAM
  • Transcend 240GB MTS420S M.2 SATA drive used for OS and varlibqubes pool. BTRFS
  • Only dom0 and sys-usb running.
  • Backup to Kioxia U301 128GB USB 3.2 stick
  • 32 qubes including 15 templates (Debian, Fedora, Archlinux, Whonix, Alpine, Kicksecure, …).
  • AppVMs are usually used for browsing (so some browser cache exists) and coding (so considerable text files exist). But not too much video or audio.
  • Total size: 110.7 GiB

Results:

Compression filter Backup time Backup size Backup verification time
gzip (default) 123m 8s 622ms 37.4 GiB 29m 5s 483s
pigz (Parallel Gzip) 52m 52s 279ms 35.8 GiB 29m 59s 242ms
zstdmt (Z Standard multi-thread) 41m 56s 15ms 36.4 GiB 32m 23s 946ms

cc @tanky0u

7 Likes

And that’s on a 2 cores / 4 threads CPU.

1 Like

We should also have the “no compression, dammit” option; I care less how big backup images are than I do that they take too damn long.
What is critical is that the installer USB image have filters installed for all choices listed in the backup GUI. It would be actively harmful to present alternatives that the latest installer does not know how to read.
Most of those in the current list and in the proposed patch are worse than pointless, as they are fatally inferior to modern methods. It would suffice to offer only “none”, “zstd” (small, fast), and “lz4” (small enough, even faster), defaulting to as much parallelism as the machine can provide. (“Lz4” might be even be faster than “none” on many SSDs.) To me this is the highest-priority improvement proposed because it affects everyone, enables hours and hours of time saved, and, most important, would cause backups to be done more frequently.

Don’t check the “compress backup” option if you do not want compression.