[Guide] Incremental backup using the official backup system

Community Guides
1

Hi,

I got really bored about backups, so I took a different approach. I like the official backup system because it works, it’s easy to use and can restore everything, however it sucks because it takes too much disk space.

Instead of fighting it, I made a simple script to replace scrypt binary in dom0 that just neuters the encryption, this allows us to pipe the backup file to a tool handling deduplication and encryption.

Here is the result of a qube backup done 3 times, with a qube restart between each:

> du -shc *
1,6G	qubes-backup-2024-04-12T112336
1,6G	qubes-backup-2024-04-12T112420
1,6G	qubes-backup-2024-04-12T112557
754M	restic-repository

The repository didn’t grow after each backup :slight_smile:

I’d be happy to receive feedback about it, I didn’t try it much yet.

Setup

In dom0, rename /usr/bin/scrypt to /usr/bin/scrypt.bak.

Create /usr/bin/scrypt with the following content, and make it executable:

#!/bin/sh

# $1 = enc / dec
# $2 = infile
# $3 = outfile

# remove parameter -f used when restoring
for arg do
    shift
    [ "$arg" = "-f" ] && continue
    set -- "$@" "$arg"
done

# stub prompt for backup
if [ "$1" = "enc" ]
then
    printf "Please enter passphrase: " >/dev/stderr
    printf "Please confirm passphrase: " >/dev/stderr
fi

# stub prompt for restore
if [ "$1" = "dec" ]
then
    printf "Please enter passphrase: " >/dev/stderr
fi 

# handle all combinations possible
# input = a file or stdin
# output = a file or stdout
if [ "$2" = "-" ]
then
    if [ -z "$3" ]
    then
        cat
    else
        cat > "$3"
    fi
else
    if [ -z "$3" ]
    then
        cat "$2"
    else
        cat "$2" > "$3"
    fi
fi

Usage

After this change, the backup tool won’t encrypt the backup anymore, you can send it to a script to handle it in borg / restic / whatyou want, I explain the method in a blog post

When making backups, just type anything in the password field, it’s not used but required for the GUI to be happy.

Make sure to disable compression, this doesn’t help deduplication, and the backup tool will handle compression anyway (and better than gzip).

Cons

You can’t use backups made prior to this change, you would have to rename /usr/bin/scrypt.old back to /usr/bin/scrypt to make them work again.

A dom0 update or upgrade may overwrite scrypt binary.

As stated in the second post of this topic, backups could be tampered with through the qube handling the backup tool. This is quite unlikely if you use packages from a template on a qube that isn’t exposed to non trusted binaries, but be sure it fits into your threat model!

Misc

I verified and dom0 should only use the scrypt command for backups and restore, so it shouldn’t break anything :slight_smile:

Performance

I made a benchmark to compare. The protocol was to use the command qvm-backup --no-compress --yes with both restic (defaults settings) and borg (compression set to zstd instead of lz4).

The default backup was done with scrypt disabled, and was using --compress, plain good old qvm-backup.

With restic and borg, I did two backups of all my qubes (except the one with the backup) in a row, then I rebooteed and did a new backup, all temporary qubes disks got merged after that, increasing the size.

I did not do the qvm-backup multiple times because the size and time would be almost identical to the first run.

Type							size (bytes / GB)	time
----
default backup with compression	94670100 (91GB)		137 minutes
restic							76439920 (73GB)		22 minutes
restic 2nd backup				77499468 (74GB)		15 minutes
restic 3rd backup post reboot	81685884 (78GB)		15 minutes
borg							78847352 (76GB)		23 minutes
borg 2nd backup					82468580 (79GB)		17 minutes
borg 3rd backup post reboot		90346032 (87GB)		19 minutes
11 Likes
2

There is a reason though why there’s currently no built-in option to bypass the scrypt utility: Assuming a strong passphrase it provides not only encryption but also integrity protection, end-to-end (where one end is the dom0 doing a backup now, and the other end is the dom0 doing a restore in the future). Without that, at least the restore VM serving the backup file is in a position to tamper with it and exploit dom0 during the restore process.

4 Likes
3

Thanks for pointing this out. That’s important to think about that depending on one’s threat model.

1 Like
4

Some performance benchmark added, with numbers :+1:

5

Just another idea - didn‘t try it myself, yet:

Depending on the hardware you have got (or you are willing to buy) you could use zfs replication with deduplication at the block level, too.

1.) setup encrypted zfs-backup-target-vm.
2.) backup your data-vms to backup-vm-datasets with qubes-backup.
3.) replicate the backup-vm-datasets to your zfs-filer (to a pool with dedup enabled).
4.) have fun with CoW, checksumming, snapshots, easy offsite replication via ssh, transparent data encryption, redundancy (depending on pool config and number of disks, that is) etc.

Downside: zfs dedup needs quite a lot of ram and slows your filer down (unless you have special mirrored vdevs for metadata/dedup tables - won‘t be lightning fast, but less slow).

6

I don’t think the restore process is practical with that? What if you reinstall Qubes OS and try to restore? Would it correctly restore everything, associate netvm and firewall rules?

7

I simplified the script, the python calls were not necessary but I used them when trying to figure why the strings were not recognized, it turned out I just had to send them to stderr, but I forgot to remove the python commands.

8

Nice solution, thank you for sharing!

I have to note that with replacing binary you fight @marmarek’s decision to force users with an obligatory encryption even when users explicitly told him (with arguments and reasons) that there are scenarios when encryption is not adding any additional security but only adds problems, like with de-duplication cases. Your solution would be much cleaner and easier if the team would listen to users and do not force such changes. often breaking things.

9

It would be nice to have a new parameter --encryption-filter like --compression-filter and cat being one valid value :sweat_smile:

10

He listened to the arguments and reasons. He also provided his own arguments and reasons. He has evidently decided that this is the best course of action for Qubes for now. That can always change in the future as circumstances change and as new arguments and reasons are considered. However, it is inaccurate to say that the team doesn’t listen to users. Not agreeing is not the same as not listening. Repeating your criticisms doesn’t make them more persuasive. If you want to be more persuasive, make better arguments.

3 Likes
11

Thank you for reply. I do not see his arguments, maybe I missed some discussion? The only place I know is here, which actually has no arguments but just a statement that something shall be done this way (with no arguments for taking this decision in 2018). So, not sure what his own arguments and reasons you mean, please send me a link if I missed it.

Sorry for criticisms, anyway.

12

Yes, exactly, it would be great.
What I am saying is that up to some version (just before R4.0) the encryption WAS optional and solutions with deduplication (like zbackup) were working well without hacks with binary replacement /usr/bin/scrypt. The update broke it.

Can this hack with replacement of /usr/bin/scrypt in dom0 cause problems in some other user cases? Maybe scrypt is used in some other scenarios, what would happen then?

13

I see two reasons there:

  1. One of the design decisions for qvm-backup in R4.0 is not to allow the extraction of unencrypted data from dom0, presumably since dom0 has access to everything in the system, which means any leaks from dom0 could be fatal. (I think you can probably find more discussion and rationale behind this design decision in the mailing list archives, if interested. Also see links below.)

  2. Encryption and integrity protection are currently handled by the same tool (scrypt), and we definitely don’t want to allow backups that aren’t integrity-protected, as that would be an extreme security risk. It seemed quite difficult to find a suitable tool that fit the requirements of qvm-backup, so switching to a different tool just to separate the encryption and integrity-protection functions could very well mean switching to a worse tool or set of tools. (In other words, changing how qvm-backup works in order to support unencrypted backups might provide some non-security benefits for some users while decreasing security for most users. I think it’s reasonable to say that this wouldn’t be a worthwhile trade-off for a security-oriented operating system.)

But that update was for a very good reason. It made qvm-backup much more secure. You can read the whole saga here:

https://groups.google.com/g/qubes-devel/c/CZ7WRwLXcnk/

2 Likes
14

unfortunately, it seems in practice everyone is hacking around their own backup tool because the one provided upstream doesn’t fit their usage :sweat_smile:

2 Likes
15

Thank you for such a large reply.

I still do not agree with the chosen design decision of the backup forced encryption approach, and I provided my arguments here on forum and on github. But I am glad that you value my opinion and provided your understanding of arguments and reasoning.

2 Likes
16

It could just seem that way because the people who don’t like the included backup tool have a reason to be vocal about it, whereas those of us who are fine with the included backup tool just quietly use it and have no reason to be vocal about it. We don’t have the usage data to know for sure. I’m just one user, but I’ve been using it daily for many years, and it serves my needs well.

3 Likes
17

It true, but the solution with optional encryption would have served you and other users in absolutely the same way, with the same level or security by default. It would also allow advanced users to get deduplication and make other tricky steps toward more sophisticated backup systems.

Currently the second group of people (I think, which is indeed much smaller but louder) is making their system way less secure by some hacks and tricks compared to the ability to make it less secure by sharing unencrypted backup data to some offline special qube for deduplcation and other actions.

So, among 2 reasons I understand only the first one:

  1. That encryption tool is also used for integrity protection. That I understand it, but it can be solved. Integrity check and encryption are not fundamentally connected, it is just the selected implementation connects them, if I get it right.

  2. The reason about not letting unencrypted data to some offline qube or even dom0 itself is not valid to me. Because by default it would have worked the same, with encryption, the GUI tool still can force settings password and etc. So, the only cases when security would be lower are the cases like in this topic, when binary is replaced in the system just to hack encryption and achieve deduplication. And all these actions are explicitly made by user. User who can run anything in dom0 as root after all.

I hope I explained my point well, because the second point should be mathematically and logically absolutely correct and, thus, convincing.

18

An elegant way to solve this which doesn’t even require any changes to the Qubes OS backup format would be to extend the scrypt utility with an option to skip AES encryption but still do HMAC-SHA256 integrity protection using the stretched password. Here’s the relevant part of the source code for scrypt enc (similar for dec):

The scrypt output format header has a version byte that’s currently always zero, one bit of it could be set to indicate unencrypted status. Maybe if someone (not me though) handed Tarsnap upstream a patch they would be open to merging that?

3 Likes
19

@solene refreshing publication, as usual. It’s nice to see some PoC showing that things can be used differently, shaking status quo. Good job at getting some eyes here and seeing some stats so that qubes backup/restore tool could be linked to other tools.

Yes…
Well, again, this is, to me, really old issue (2015!) [Contribution] qubes-incremental-backup-poc OR Wyng backup · Issue #858 · QubesOS/qubes-issues · GitHub

And I do not see, really, how wyng-backup could get better but if more eyes were on it. Don’t get me wrong, qubes backup/restore has its use and I still use it personally once in a while, but I do not see how GitHub - tasket/wyng-backup: Fast Time Machine-like backups for logical volumes & disk images and GitHub - tasket/wyng-util-qubes: Qubes integration for Wyng enables backup and restore by VM name doesn’t solve all the issues here while going even further with incremental backups that can literally run under a minute, not 15 (restic) or 17 (borg) .

Of course, i’m mixing a lot of things here as usual, but with bees+brtfs+wyng-backups, people could backup/restore all cloned+derived/specialized templates created by qusal/shaker, while brtfs alone could be extensively used to restore a local (and at no cost) snapshot that was taken 30 days ago (just like one of your post @solene), that being without external backups.

But to arrive there, yes, we need to, on one side, show the limitations of currently offered tools, but also most importantly prepare the next ones that fit better what the ecosystem is using those tools for. Anyway, this is my opinion and I don’t expect everyone to understand until those tools are packaged/distributed under contrib repositories…

But this thread again has a lot of technical people’s eyes, so I take the opportunity to remind people that qubes backup/restore has evolved a bit, but is mostly as it was designed years ago so people can backup/restore whole backups archives. If wyng-backups was criticized against lack of encryption/authentication mechanisms in the past (my 2021 presentation) , it was also resolved by proper implementation while [Contribution] qubes-incremental-backup-poc OR Wyng backup · Issue #858 · QubesOS/qubes-issues · GitHub is still open and ink is spreading again on limitations of qubes backup/restore tool… Why are we not replacing/extending it by something more useful made to do grand things at his inception and improving since then? If before, wyng was not integrated with qubes api to get vm dependencies/properties and was just a block level incremental backup tool, with the arrival of wyng-util-qubes, user can restore a qube+properties just as qubes tool.

I guess my question now is: what is missing for wyng-backup/wyng-backup-util to deeply be considered for what it offers and what involvement is needed for it to replace/extend qubes current backup/restore limited functionalities?

@adw : maybe clarify next steps needed either

4 Likes
20

That’s not true (or at least not known) for reasons already discussed above.

If I understand your reasoning here, you’re saying that since users can (with enough tinkering) do whatever they want in dom0, they can still extract unencrypted data from dom0. Since users can (with enough tinkering) extract unencrypted data from dom0, then the Qubes backup tool should not try to prohibit the extraction of unencrypted data from dom0.

However, this reasoning can be deployed to argue against any and every security goal, including fundamental ones like “an untrusted web browsing qube should not be able to compromise an offline password vault qube” or “an app qube should not be able to compromise its parent template.”

For example, since users can (with enough tinkering) do whatever they want in dom0 (or edit the source code and compile their own ISO from source, by the way), they can also destroy the compartmentalization between VMs. Since users can (with enough tinkering) destroy the compartmentalization between VMs, then it follows (by your reasoning) that Qubes OS itself should not try to enforce secure compartmentalization between VMs. However, that would be absurd, since enforcing secure compartmentalization between VMs is an essential, fundamental, and primary feature of Qubes OS.

The flaw in your reasoning is thinking that since it’s possible for users to intentionally do X, Qubes software shouldn’t try to prohibit X. But that’s like saying, “Since it’s possible for users to intentionally cut themselves with razors, safety razors shouldn’t exist.” There’s a reason that many useful tools come with built-in safeguards. Most people who use the tool wish to use it for its intended function. The safeguards ensure that the tool serves its intended function while preventing harm when the tool is used in unintended ways.

Qubes is open-source software that prizes user freedom, so it allows users to tinker with things in ways that many other mainstream OSes do not, but this doesn’t mean that it shouldn’t make an effort to protect users from shooting themselves in the foot.

4 Likes