[Ideas] Why do people not make backups? How to improve that?

General Discussion
21

  1. Most people never really think about backups before they experience the pain of losing some data that’s important to them. This isn’t specific to Qubes. It’s human nature. The phenomenon has been widespread since the birth of computing.

  2. Some people know they should make regular backups but are too lazy/forgetful/time-constrained/mental-bandwidth-constrained to do so.

Automatic backups would help both groups, while refinements to the current backup tools would help the latter group but not the former.

FWIW, the current backup tools are already in the core documentation.

I let it run while I’m doing other things or sleeping, so the effective time cost for me is just a few minutes.

This example doesn’t show that these tools fail to create adequate backups of data. Rather, it shows that the time and effort required to restore data with these tools are too high for this particular CEO’s needs. As you know, there’s an inherent trade-off between security and convenience. Part of the reason that the Qubes backup tool is less convenient is because it’s more secure. It allows you to safely restore from a backup even after an adversary has had access to that backup, which is a feature most other backup options lack.

They can’t see any of your files, because Qubes backups are always encrypted.

No, Qubes backups have built-in, automatic integrity checking.

You can write a simple shell script to handle this for you.

You can select certain qubes to be backed up by default, then you don’t have to choose them manually. You can also just have a text list of qubes, then do:

qvm-backup [...] `cat list.txt`

If you want, you can use the --passphrase-file option.

Don’t forget to verify (i.e., test restore) the backup! As the saying goes, “Backups always succeed. It’s restores that fail.” A backup is useless if you can’t restore your data from it.

Calculating hashes is not necessary for security (see above), but it can be useful for organizational purposes.

Statement is too strong to be accurate. You can aim for something without hitting the target. Whether any given piece software is “usable enough” is a matter of opinion and depends on the specific criteria of evaluation.

4 Likes
22

To answer to the main question:
I have done a single backup with checking and so on, it worked fine. But in the long run, I also notice that I make far greater breaks between backups, compared to my previous debian system with timeshift, even though I had multiple occasions where I lost sensible data.

The reason behind this is probably the large amount of time it takes to figure out how to create a meaningful backup. I have also never really recovered with a backup, only checking it on a working system.

I have two questions:

  1. Is it alright to use the qubes that are being backed up at the same time?
  2. Just out of curiosity: Why wouldn’t it work to launch a live system with a usb drive, decrypt the qubes partition and make an incremental backup of the qubes partition :wink: ? I can’t think of a definite reason not to do so, but I can imagine that more technical people know a pretty good one…

Thank you!

1 Like
23

Statement is too strong to be accurate.

A statement’s validity and accuracy come from the facts supporting it. In the particular case:

  • Is there an official stable incremental backup system in Qubes? - No.
  • Are there people desiring that? - Yes (in various threads).
  • Is there a flexible configuration such as that of Bacula? - No.
  • Is that good usability - No.
  • Are there people willing better UX of the backup process? - With a major priority:
  • Is there also a major long-standing and unresolved issue with backup potentially not actually backing up?

These are all facts.

You can aim for something without hitting the target.

Indeed. But point is that the practical value for the end user comes after hitting it.

Whether any given piece software is “usable enough” is a matter of opinion and depends on the specific criteria of evaluation.

I did not use the quoted phrase, just answered the OP’s question based on my personal experience.

2 Likes
24

“adequate backups” - an interesting phrase.
Taking a full disk image using dd is an “adequate backup”, but almost
entirely useless for most people’s needs. You have to consider the
utility of a backup in terms of what is being backed up, and how it
will be restored.
In my opinion, (as you should know by now), Qubes makes a mistake in
focussing on the qubes. Most users should care less about the qubes, and
more about the stored data.
In my experience providing support, I have rarely been asked to restore
a full system. I have very often been asked to restore a single file,
to restore a lost directory, to find a particular version of a file,
etc. Is there something about Qubes that changes these requirements? I
dont think so. The idea that you have to restore 300GB of data to access
one file strikes me as, well, odd.

I never presume to speak for the Qubes team. When I comment in the Forum I speak for myself.
4 Likes
25

I agree, I came up with wyng to backup all my qubes and also a restic script deployed through templates to backup /rw per qubes on a remote repository.

I can restore all my qubes if I need to quickly recover and I can also restore per file, or browse the backup repository using a fuse mount.

The first step could be avoided with 100% automation, but I gave up with salt for my personal use. Almost everything is salted, except a few parts that were too annoying to automate.

26

If you read this topic and you are not doing backups, please tell us why and what would make you create backups

Right now, I have a working Qubes system, and I can use it reasonably well. However:

  • I still find Qubes OS a bit of a mystery box
  • I am more focussed on trying to understand Qubes OS, I don’t have any meaningful data to back up

Given the sensory overload that a new user will face, the system will need to be graphical and very simple. I can handle updates to Qubes precisely because it is graphical and very simple. Mouse on over to the update icon in the top right corner, run the updates, close when done. I can handle this.

I also have a system running Arch Linux, with a BTRFS filesystem. When it updates, I have inserted a hook which automatically does a snapshot, so I can roll back if the update fails. Somewhat annoyingly, I appear to have a systemd menu rather than a grub menu, so I cannot select a snapshot at boot time. I guess I’m hoping to make it as far as the timeshift tool so that I can roll back. It’s a work in progress.

27

Qubes OS provides a similar process, although it is not a backup, everytime you start a qube you can revert its storage to the boot before (by default it keeps 1 snapshot but can be configured).

This is really handy when you screw everything in a gentoo template, or when you did a big mistake in a qube.

1 Like
28

All interesting. I do NOT lilke the idea of marking qubes based on when they were last backed up. The reason? Several:

  1. Almost every template on my system is automatically generated. If I lose it, I can just regenerate it with less than two minutes of time at the terminal. (The exception being my Windows 7 template.) So I do not need to back them up ever, and don’t want to be nagged to do so, nor do I want to see a “last backed up on” field for these because that’s about like seeing a “last oil change” sticker on an electric car.

  2. Almost every appvm is really a disposable based on a dvm template that is also autogenerated, and can be rebuilt as part of the same process that rebuilds the template. I don’t need to back those up either; same comments apply.

  3. There are a handful of regular AppVMs (i.e., ones that aren’t dvm templates) on my system, THOSE have actual data on them and do need to be backed up but in many cases I don’t access them daily. I actually gimicked things such that when these VMs run, they call a service that leaves a “marker” on dom0; at 3 AM a cron job will look for the markers, and, if the VM isn’t running, back it up and delete the marker. That way they get backed up only when it’s possible they have changed.

  4. Dom0 has the info needed to regenerate all those qubes; I do back that up (as well as keeping it in a “repository” AppVM that is one of the ones I back up). (And note: dom0 backups ONLY copy your home directory. So I wrote a script to copy that info out of /srv/user_salt and /srv/user_pillar into my home directory before backing up dom0.)

  5. That leaves data. Data lives on a NAS (and gets mounted on the disposable VMs), so that gets backed up outside of Qubes, again completely automated. If it weren’t on a NAS, I’d have it on AppVMs which would work like case 4; indeed that’s why I back up my AppVMs. (The NAS is handy; it’s one destination for backups, another is a second SSD not mounted to QubesOS in the computer itself.)

So backup is critical, but I have minimized the amount that gets backed up. And, I suspect, minimizing the amount that Qubes must back up would help people do it more; the case Solene mentioned of someone needing to back up 800GB would be a nightmare.

2 Likes
29

In my opinion, (as you should know by now), Qubes makes a mistake in
focussing on the qubes. Most users should care less about the qubes, and
more about the stored data.

Exactly.

The idea that you have to restore 300GB of data to access
one file strikes me as, well, odd.

Or to backup daily (which is the more frequent process) e.g. a full 10 GB VM just because you have worked on a few files, totalling 200 KB. Consider also the storage overhead if one wants to keep e.g. last N versions of each backup.

30

Agreed, but your statement implies that Qubes doesn’t aim for usability. The facts you listed don’t support the contention that Qubes doesn’t aim for usability, only that it doesn’t achieve usability (in your opinion).

Not that I’m aware of. (No, #3498 doesn’t fit that description.)

Sure, but that’s not the statement in dispute.

You didn’t use the quoted phrase, but did use the phrase “goes against all claims that Qubes aims for usability,” which clearly implies that you think Qubes falls short of some acceptable level of usability. “Usable enough” is just intended to be a shorter way of saying “achieves an acceptable level of usability.”

It’s your phrase:

There’s already a graphical backup tool similar to the graphical update tool.

31

Agreed, but your statement implies that Qubes doesn’t aim for usability. The facts you listed don’t support the contention that Qubes doesn’t aim for usability, only that it doesn’t achieve usability (in your opinion).

The word intelligence comes from inter (between) and legere (read), i.e. reading between the lines.

What I said (what you called a statement):

As long as there is no decent (official, stable, not beta/testing/someday) incremental and differential backup, fine-tunable etc, that goes against all claims that Qubes aims for usability too.

This does not mean “Qubes does not aim for usability at all”. It means that the current state of the backup tool and the lack of observable improvement in the last 2 years don’t show alignment with the goal, as it is not only sub-optimal usability-wise but also lacks the mentioned important functionality, i.e. the problem goes even beyond usability. This is not something that I believe or fantasize about but an observable fact.

Your own words:

As you know, there’s an inherent trade-off between security and convenience. Part of the reason that the Qubes backup tool is less convenient is because it’s more secure.

confirm that the usability is not good as it could be.

If I may point out, the mentioned trade-off does not really apply to the current subject, because encrypting 200 KB of data in an incremental session is not less secure than encrypting 10 GB in a full backup, ceteris paribus.

Not that I’m aware of. (No, #3498 doesn’t fit that description.)

OK. I misread that one, sorry. It still is a long-standing usability issue though.

32

I know. I was attempting to flesh out what I meant by an adequate
backup, and why I think that Qubes, Wyng, and dd are not adequate. I’m
sorry if that was not clear to you.

I never presume to speak for the Qubes team.
When I comment in the Forum I speak for myself.

33

@unman

How do you backup?

34

So either you do not use the graphical UI for backups, or you do not verify your backups. In the UI, the latter requires to start over the whole procedure again in 5 hours, which is extremely annoying and why I make backups less frequently than I should.

Which is why for most users this is too inconvenient. I’m fine with the command line, but even I can’t use it for everything. It takes time to learn new commands (there’s always too much to learn) and new users won’t be able to start doing it immediately, loosing precious time and risking their data.

This tells more about your specific users than the general situation. Backups of whole qubes helped me many times when qubes stop booting or I want to clone my installation to another, independent device. Also when I upgrade to a new Qubes version. I love that I don’t have to make any in-qube configuration and all installed software just works immediately after the restoration. The disk space overhead is tiny and the restoration process is straightforward. I’m happy that Qubes chose this direction by default.

Here it is.

4 Likes
35

I really like that Qubes doesn’t only focus on user data but Qubes.

I have a lot of extremely configured Qubes and I take great comfort in the knowledge that they will work on any Qubes system I come across.

I won’t deny that by default user data should be the focus but I beg you to not depreciate the ability to backup whole Qubes.

5 Likes
36

Sorry for the noise. This topic’s overdoing. I mean, it looks it has too many do’s in the subject? How it can be read like this, it looks like it suggest people not to make backups?

37

Not necessarily. Due to the inherent trade-off between security and convenience, it’s possible for a given piece of software to be simultaneously:

  1. As convenient as it can be without sacrificing security (i.e., the only way to make it more convenient would be to give up some security), and
  2. Less convenient than other options (which are able to be more convenient because they aren’t as secure).

(To be clear, I don’t think the Qubes backup tool, or really any piece of software in the real world, actually achieves this.)

No, that’s not the case. For example, sometimes I:

  1. Start the backup.
  2. Go do other stuff.
  3. Come back and see that the backup is done. Start the verification check.
  4. Go do other stuff.
  5. Come back and see that the verification check is done. Do whatever else I need to do with the backup.

Steps 1, 3, and 5 combined don’t take that long, so the total time I actively have to spend on the backup process isn’t that long.

(To be clear, I do think there’s a lot of room for improvement here.)

Well, at least the verification doesn’t take nearly as long as the backup itself, so it’s not quite as bad as starting the whole procedure over, but I’ve long thought it would be more convenient to have an option to automatically verify the backup file upon creation.

I think this is different from what he meant. I think he meant something more like “a bug in the backup process that sometimes causes random data to be missing from the backup.” By contrast, this issue is more like, “X isn’t included in backups by default, but users would benefit from it being included, so we should figure out a way to start including X by default.”

1 Like
38

Unless if it’s Win qube with Xen PV win disk drivers installed. There’s no backup that will work here. And will not any time soon, if ever.

39

Why? The Qubes OS backup tool works at the volume level, it does not care what’s inside :thinking:

1 Like
40

Its so time consuming
Backing up Qubes often means connecting drive and leaving backup program running for many hours.

I am not confident about using Qubes during a backup and I don’t want to be online while using all VMs. I worry about being compromised through an attack vect or while backing up that will make it easier to open up backup

Qubes has no option to just back up important data for smaller amount. Wish I had is for Qubes to backup only data in templates that is needed to run installed programs. Backup of many GB gets expensive for poorer Qubers

I end up backing up templates because I don’t remember what I’ve installed in everything and it takes so much time but would be quicker if Qubes created a backup list for everything installed in each template so it could be reinstalled through script in a worst case scenario of major data loss… Backing up in Qubes either requires lot of storage or requires lot of thinking to determine what to leave out. It’s an exercise in stress and thinking each time.

I would like home folders copied, crucial data copied, and list of Apps from each template and the names of each template copied. I don’t need everything I just need to be able to back up in an emergency. There’s no way to just do home data only backup of some VMs quick

There’s no way to create a cronjob in dom0 to automatically backup. There’s no command line method of running backup can’t make script and run with cronjob in dom0. There’s no Qubes TimeShift in Qubes and trying to manually install Timeshift to dom0 could not be goods

Would like 2 types of cronjob backup. 1 short backup every week of home of most VMs and App list through dpkg or other way and 1 cronjob for large backup of all data every few month but not way to do

3 Likes