I think @solene had a forum discussion thread in the past, something along the lines of, “why aren’t qubesos users do frequent backups?” Well this is it, my answer: on my QubesOS 4.2.4 system, whole system backup takes me 9 hours to complete. The backup size is around 600 GB. I can never simply casually start my backup process and let it be finished in couple of hours. Nope. I have to dedicate a whole night to the backup process, and I have to be especially mindful about when to start the backup (I cannot start the backup 2 AM in the night, otherwise the backup process bleeds into the next day, forcing me to abort the backup process in order to pack my laptop and go out).
Q: Why don’t you selectively backup, instead of backing up your whole system? A: Well this is my system and I want to backup all my qubes. However, I am actually leaving out a qube of mine which has 1 TB worth of movies and TV Shows out of this “600 GB backup”, so, I think I already satisfy your “backup selectively” suggestion.
I am currently using Samsung 870 QVO (1TB) external SSD drive for backups. It gets connected to my laptop’s USB-3 port over SATA-to-USB converter dongle. Is my SSD speeds to blame? If I get a new 4 TB NVMe SSD drive, will my backup times get reduced to ~3 hours MAX?
We can discuss general tips and tricks to speed up the QubesOS Backup process here.
If in dom0 top you see that gzip process is running at 100%, then compression is slowing down the backup process, using a compression program supporting multiple CPUs might remove this bottleneck.
However, you will quickly get bottlenecked by scrypt that does the backup encryption, and it is not multi-threaded.
You could switch to wyng backup tool, it only saves the difference between each backup, so the process would take a couple of minutes running it daily. I have no certainty, but I think it will become Qubes OS default backup tool in the future.
This is interesting, as I always enable “Compress backups” option in QubesOS Backups GUI program. I didn’t know gzip was a “single-core” thing (is this like emacs?), sucks to see, if indeed it is causing this bottleneck.
Yet another one; why’s GNU-land not-privy to writing/using multi-threaded programs?
Never heard about this before. However, I am anxious about using a backup tool other than what the devs “officially” provide, as backups in QubesOS are already something out of the ordinary, due to QubesOS’s own idiosyncratic system architecture.
It highly depend on what you have in your qubes of course, if you have highly compressible data, it will be a lot faster as it will be less data for scrypt to encrypt.
That is correct. If using Qubes OS r4.3, I would recommend installing zstd in dom0 and it will add zstd and zstdmt (multi-thread) options to the backup dialog. Speed difference is like day and night. Not even close. Even on my dual core/quad thread system. For Qubes OS r4.2, it is possible do use zstd compression via CLI backup tool.
While I am not personally a member of Qubes OS core development team, I have been in touch with them and discussed the future of existing traditional backup tool and Wyng. Based on feedback I have received, Wyng will be an integral part of Qubes OS in the future. It is developed by a well reputed and well respected developer who has had close collaboration with Qubes OS team for a long time.
Having said that, I believe both solutions are good. Traditional backup is still nice since it provides compressed & encrypted backup in a single file which could be stored in most ordinary filesystems. It is excellent for full-backup scenarios.
Can you give an anecdote from your experience. How many hours was your backup process taking before using zstdmt and how many hours is it taking now with zstdmt?
Sure. I have to do my weekly backup. I will measure difference this time. But please note that my main drive is small (240GB). And only 150G is used. And the backup medium is Kioxia U301 128GB (USB 3.2). I will post the results on Solene’s forum thread
Support for zstd (as well as other auxiliary compression filters) is integrated into the core and GUI tools. The only thing you have to do is to sudo qubes-dom0-update zstd in dom0 to install it.
Some more notes on zstd (Z Standard).
Vanilla Fedora installations has been enabling it it as their default compression filter on their default filesystem since Fedora 34. Details here.
Fedora 43 will use it as the default Linux Kernel initramfs compression. Details here.
Fedora changed the default RPM (package) compression to zstd since Fedora 31. Details here.
Archlinux switched to zstd for package compression since 2020. Details here.
Debian is going to swtich to zstd for package compression. Ubuntu has already done so. Details here.
I use a totally different approach…and backups take mere minutes for me even though I have 40-odd templates.
It’s several different things in combination, and even if you don’t do them all any particular one might help.
First off most of my data is on a NAS, in encrypted containers. THAT gets backed up in the AM automatically–it first does md5sums to see if the containers have been altered. That happens totally off the qubes system so in this context it’s “free.”
That leaves the few qubes that do have data on them (including dom0, which has a lot of stuff to do with system configuration), and I back those up at 3 AM. (Note that backing up dom0 only backs up a few directories; my backup script has to copy other things (like my salt states, and stuff I have in /usr/local/bin) into those directories before proceeding) My shortcuts that start these qubes all mark that the qube has been accessed; the backup script checks to ensure that the qube has been accessed and that it’s not running yet.
OK, but what about all those other qubes? Wouldn’t it be a pain in the butt to have to rebuild them from scratch? No, actually. Everything is configured by Salt–it leaves alone AppVMs with data on them but rebuilds all templates, other AppVMs, and named disposables. The salt configurations are kept on a special AppVM (which gets backed up) and to a thumbdrive.
So if my system went belly up and I lost everything, I’d have to reinstall QubesOS, restore that special AppVM or use the thumbdrive, then via Salt rebuild all my other Qubes (then restore all of the short list of VMs that are backed up). That would take time, but it’s unlikely to be needed so a lengthy recovery time is acceptable to me.
Also noticed TailsOS 7.0 switched to using zstd from xz:
Faster startup
Tails 7.0 starts 10–15 seconds faster on most computers.
We achieve this by changing the compression algorithm of the Tails USB
and ISO images from xz to zstd. As a consequence, the image is 10%
bigger than it would be with the previous algorithm.
I want to point out that if your qubes are that big, you probably don’t actually want compression. Not everything is compressible, and it’s a mathematical fact that all reversible compression algorithms may generate an output bigger than the input. Videos, pictures, sound files (or generally file types that are already compressed) will all waste CPU cycles with an output size very similar to the input size. Encrypted files will do the same (unless encoded in something like base64), but is also almost guaranteed to end up bigger after compression.
(Note: I’m making the assumption that Qubes doesn’t differentiate between files that are easy and files that are hard to compress)
The Qubes Backup and Wyng don’t differentiate between files at all,
because they are not file based systems. They work with volumes.
This makes them unsuitable for many normal backup/restore operations.
Want to find a file that you deleted some time in the last month? Restore every backup that you took, and look through the volumes til you
find it.
Want to find a specific version of that file you were working on? Again,
restore every backup and check until you find it.
Note that if you have also added a large quantity of other data in the
meantime you have to restore all that data until you find the one
file you are looking for.
No doubt the Qube Backup, and Wyng serve a specific use case. It’s just
that in my experience it isnt what most users want from their backups.
dd’ing a disk is undoubtedly a good way of preserving data. But it
doesnt serve most people’s expectations of a backup/restore process.
Qube Backup and Wyng - the same.
You should be aware of these limitations and be happy to work with them:
otherwise, put another backup regime in place that will serve your
needs.
I never presume to speak for the Qubes team.
When I comment in the Forum I speak for myself.
