Incremental backup possibilities

As awesome as Qubes is, really the only thing that holds me back from using Qubes as my daily driver is lack of incremental backups.

On my current Linux machine, I use automatic borg for very fast backups (about 2.5 minutes for 500GB incremental backup, if little to no changes since last backup) and I can’t imagine not having that peace of mind now on another OS.

The native backup tool is blind to changes, and very slow - for me it’s about 1GB of data per minute, which means to backup 500GB of qubes you need over 8 hours!!

Obviously, running such a long task daily is painfully impractical. Even weekly that requires some planning as you need to leave your machine running overnight for it to finish. Also, keeping older backups is difficult because they take that much more space, with no incrementality.

There is an issue open since 2015: [Contribution] qubes-incremental-backup-poc OR Wyng backup · Issue #858 · QubesOS/qubes-issues · GitHub

Wyng is now in beta.

Some questions:

  • Is Wyng production-ready in its current state? As in, could I theoretically rely only on backups created with it, with disaster (hard drive failure) recovery in mind?
  • Probably no one can answer this, but if Wyng gets its first stable release, will we get official integration ASAP? How high is that on priority list?
3 Likes

The “incremental backup” issue is mentioned many times in this thread:

Maybe you want to add your question there… although the chances of it being answered are close to zero.

1 Like

Dont let this hold you back.
The backup tool is good for backing up a qube with its configuration in
a secure way.
But most of the time your concern is with the data, not the qube that
holds it.
You can run any of the tools you usually do, including borg. I use
zpagfranz for incremental backups,(against the snap volume for running
qubes), because I’m used to it, but you can use anything you like.

Optimally, use a disposable for the backups, and attach the relevant
volumes, rather than run anything in dom0. Or you can use syncthing to
keep data synced between your data producers/consumers and your backup
system.

Personally I tend not to worry about the templates or qube configuration
because my systems are fully salted, so I can recreate template and
qubes at will, but you may prefer to hold Qubes backups of those things
as snapshots, with incremental backups of the data.
Focus on the data and you’ll be fine.

1 Like

I was hoping to implement incremental backup myself for my scenarios, but the option to make backups without encryption was removed for no valid reason. Even if I make backup to the special dedicated completely offline qube, with a strongly encrypted veracrypt volume mounted, I still need to make backup with additional unwholesome encryption that:

  1. slows down the process,
  2. consume additional power,
  3. makes recovery process more complicated,
  4. makes it more difficult to implement incremental backups,
  5. … and adds 0 (zero) additional security.

This was done several years ago with an argument that incremental backups will be implemented in some future. We are still not in that future, unfortunately.

I am not sure Qubes OS even has a real priority list or roadmap nowadays. It’s one of the things that I would like to see in the development process.

2 Likes

I can’t find this in search engine. Misspelling?

Wouldn’t that totally break the isolation of Qubes? All qubes would have to talk to another qube that hosts all data from everywhere.

How do I do that? Do you mean the LVMs from the domu pool?

is Salt something like Ansible? can I generate salt from my existing qube definitions?

zpaqfranz

This is what I do: I have an external drive with a LVM volume for each qube I want to backup + one for the Qubes backup tool (I still use it for some templates and qubes). I attach this disk to a qube in order to decrypt it. Then, I can attach any LVM volume to its corresponding qube, mount it and do the backup (I use borgbackup).

Of course, the qube decrypting the drive could have access to all the data … I’m not concerned about this right now, but maybe a better solution could be this multi-layer encryption tool called qcrypt: GitHub - 3hhh/qcrypt: multilayer encryption tool for Qubes OS

As for Salt, I don’t think there is a way to generate salt states from templates. It requires more time but will give you a better understanding of Qubes OS. Leo wrote a good introduction there: Qubes Salt Beginner's Guide

2 Likes

Indeed
Thanks

Is building from source the only way to get this into a template?

Source code located in a single 3,4Mb file containing some base64 encoded parts doesn’t inspire confidence to me. Maybe am I too lazy to read this whole one-man-made code to audit anything ?

Cherry on the cake, a persuasion-filled readme instead of a factual summary.

I don’t say it wont work, neither do I say any evil is located inside. As the Russians say, trust me, but check.

restic or borg could be used in place of zpaqfranz anyway, they support raw data if you want to send a LVM partition to it, it should work with deduplication. People use it to save VMs.

If it were in 100 35k files would you be more likely to read it? :slight_smile:

Cherry on the cake, a persuasion-filled readme instead of a factual summary.

It does work - if that’s too much you could always use the original
zpaq, which is pre-packaged for many distros.

There are some pretty big usability problems with this:

  • On every boot, you have to manually attach every LVM to every qube
  • All of your qubes must be running to be backed up
  • There is no good way to backup non-standard RW dirs outside of /home aside from hardcoding them to your backup script, and remember to always update it whenever some change

So theoretically, would it be possible to install borg in dom0 and write a script to auto-detect all domU LVMs and automatically backup them to some external drive? Also, that wouldn’t include qube settings itself, yes?

1 Like

yes

right, which isn’t practical :confused:

1 Like

Anyhow, this appears to be the least convoluted, fast and reliable incremental backup for now:

  • back up all domU LVMs, and dom0’s /home, with borg
  • salt all of your qubes and recreate them with salt on restore
    • you should take notes on what you do in templates anyway, so you can quickly recreate them if needed

how do you restore though? :thinking: I would have to manually create all LVMs first?

it could be used to restore files, but not qubes themselves, this is not practical as pointed out

1 Like

Yeah, not an “incremental backup” solution by a huge margin…

1 Like

my backup scheme is:

  • i have a backup script in my template, I daily run it on qubes I use, it send /rw/ to a remote restic repository
  • sometimes I generate qubes os backups and save it on 1 local and 1 remote servers

The first solution allows me to recover files I would have deleted

The second solution allows me to easily restore qubes os if something went wrong

If I was able to do the two at once, that would be definitely better… But it’s already good enough.

2 Likes

I heard that both Pop!_OS and NixOS have made it to where one may copy the system’s image and individual packages as well as the config even to backup and replicate like MacOS “time machine” and even backup and replicate the same “setup” install throughout a network.

I am unsure what the exact technical coded details are of both, but if users want incremental backups of Templates and qubeVMs added to QubesOS then maybe the QubesOS dev team can study the schematics of how both Pop!_OS + NixOS does it.
(I am not at all advocating for a MacOS like “Time Machine” as this would increase attack surface, but a backup pushed into a Vault VM or to an external USB drive with packaged container modules that NixOS could accomplish seems implementable in replicating the same method in a way that may work for QubesOS by borrowing some of the NixOS concepts and some from Pop!_OS too)

Example references:

Skip to 10 minutes and 50 seconds into the video on the “restore”/“backup” of Pop!_OS

Skip to 2 minutes and 16 seconds (end at 3 minutes and 48 seconds) into the video on the “reproducible” NixOS file system architecture which makes it seamlessly reproducible to install on multiple workstations across a network for a team or organization by running one command after updating a single config file declaring the settings to be set and what “packages” of apps to include :wink:
(I am suggesting possibly seeing if this could be a solution to implementing an incremental backup for Templates and even full qube VM backups too in a secure isolation compartmentalized way, like :slight_smile: in a secure QubesOS type of way)

1 Like