How do you organize your backups?

Before I discovered “Qubes OS” (it’s smart now I’m addicted), I was using Debian and my backups were done with rsync on two external media and with “duplicati” to a remote server. The advantage is that my backups were incremental and I didn’t have to do a full backup every time.

If I understand correctly, I have to make a full backup (of each vm) each time, which can quickly become time consuming if my vm occupy a lot of space.

I would like to have your point of view on how you organize your backups.

You don’t need to make a full Qubes backup every time.
The Qubes backup mechanism backs up the qube configuration and the
data that it contains.
There is no reason why you should not back up just the data using
whatever means you are comfortable with.

Since you have used rsync in the past, you can use that now.
You can attach a USB device to the qube, and rsync to that.
If you want to rsync to another qube, you could use an offline qube to
capture backups from many qubes - take a look at
GitHub - unman/qubes-sync: Simple syncin between qubes over qrexec for a suggestion of how you might
do this.
You can then write off the data archive to external media, or use an
online qube and duplicati.

Once you have this in place you can also run full Qubes backups when
you like.

From Backup docs

A standard recommendation is to make backups at least weekly: three copies in two different formats, one off-site.

I am curious how others do the ‘two different formats’. I will continue to use the Qubes Backups app because it’s to two local external drives and the lack of the de-duplication is not an issue for me, but for security I would like a second format as well. I was planning on also using rsync (or perhaps Restic which has de-duplication + good encryption) but it’s not clear to me how to carry this out for several Qubes, rather than manually making the backup from each Qube which would be far too time consuming to do weekly. Presumably rsync/restic would need to run from dom0 - how how would qubes be selected?

Edit: Just seeing Unman’s qubes-sync which looks like it answers this question.

Wyng is an incremental disk image backup that can handle VM data in a fraction of the time it takes rsync, restic, etc. without breaching Qubes security model. I’ve been working on it for a couple years and its been very reliable.

What is different about Wyng is that it uses data change information (deltas) that’s already available to the system (specifically, LVM) so it doesn’t have to scan the source data for changes each time a backup is started; Wyng knows what has changed pretty much instantly. It can also perform de-duplication to reduce the backup size.

One thing I’ve noticed about file-based backups (like rsync) on Qubes is, besides their inherent security risks, they also take a gamble with data corruption when they use file timestamps as a way to shorten backup times. This is because Qubes VMs don’t track the overall system time accurately, and the clock synchronization system Qubes uses is less than perfect; If a VM’s clock snaps backward, files will be written with earlier timestamps.

So overall, the most Qubes-like approach is to backup at the disk image level; it is just tossing blocks around and not interfacing with server or VM filesystems so there’s less risk. This is what Qubes Backup does, except it sends only whole disk images. Wyng asks Linux LVM which bits of a disk image have changed and sends only those blocks, adding them to the earlier blocks in the backup archive.

3 Likes

Incredible! Do you use this as a second format to Qubes Backup, or exclusively? I noticed Cryfs is mentioned as an encryption option - do people have thoughts on how it compares to gocryptfs for this use case? Is such an encrypted volume + LUKS overkill you think?

“Format” here typically refers to the physical medium on which the backup resides. Here are some examples of things commonly regarded as different formats: external hard drive, cloud storage, flash drive, optical disc, tape.

1 Like

Wyng is my primary choice for everything. I use it on my non-Qubes systems also, as long as they use LVM thin pools for storage.

Yes, it would be overkill to combine an encrypted filesystem with LUKS. The Wyng notes mention using LUKS (or dm-crypt) for Qubes, as this puts every sensitive aspect under dom0 control without exposing dom0.

OTOH, an encrypted filesystem like gocryptfs would work, but its mostly compromises compared to using LUKS.

1 Like

Thank you for all these recipes, I will try to concoct mine from yours :slightly_smiling_face:

I am really excited to use Wyng once I understand LVM, but haven’t gotten there yet, so I’m looking for a noob friendly option in the meantime.

I was wondering what people think of the Qubes backup advice given at anonymousplanet.org - use clonezilla while the machine is powered down: The Hitchhiker’s Guide to Online Anonymity | How I learned to start worrying and love privacy

Regarding solutions like Qubes Backup:

These backup utilities will not be able to restore your encrypted drive as-is as they do not support those encrypted file systems natively. And so, these restore will require more work to restore your system in an encrypted state (re-encryption after restore).

Regarding the offline clonezilla proposal:

This backup will back up the encrypted disk as-is and therefore will be encrypted by default with the same mechanism (it is more like a fire and forget solution). The restore will also restore the encryption as-is and your system will immediately be ready to use after a restore.

Regarding Qubes specifically:

Qubes OS recommends using their own utility for backups[…]. But I think it is just a hassle and provides limited added value unless you just want to back-up a single Qube. So instead, I am also recommending just making a full image with Clonezilla which will remove all the hassle and bring you back a working system in a few easy steps.

  • Would restoring using this clonezilla method be potentially problematic? I don’t know enough about LUKS headers etc to know.
  • Is there a benefit to Qubes Backup that they aren’t considering? It’s not clear to me what ‘added value’ they reference actually is.

One of the main advantages of the Qubes backup system is that the backups it creates are not only encrypted but authenticated (and thereby also integrity-protected). Most other backup methods, including DIY procedures, do not feature this property or do not implement it correctly. (For example, if you’re not sure whether you should encrypt-then-MAC or MAC-then-encrypt, or if you don’t see the difference, then you should not be taking a DIY approach to this if security matters.) Authenticated backups protect against attacks in which an adversary, unbeknownst to you, maliciously modifies or replaces your backup and, upon restoring from that backup, you compromise your system.

2 Likes

It should also be said that taking a full disk image is not in any sense
a useful backup in Qubes. Yes, it will allow you to restore the system at
a specific point in time, but that doesn’t make for a good backup regime.
Regardless of the time involved, and the multiplicity of cloned drives
needed, it is really not suited to the problem.

How often will you be making that disk image?
What will you do with old images?
Will you take a new clone every time you update dom0 or a template?
Every time you do some work?
Really??
What happens if you discover you deleted a crucial entry in KeePassXC
4 months ago?

Clonezilla is a great tool for disk imaging, no doubt. But I don’t see
full disk imaging as particularly useful in Qubes, and certainly not to
replace qube and data backups.

4 posts were split to a new topic: Backing stuff into a qube; then backing up that qube?