Backup Fails since upgrade to 4.1 (unless qubes are all shut down)

User Support
6

No, good idea.

https://dev.qubes-os.org/projects/core-admin-client/en/latest/manpages/qvm-backup.html

Got it running. Same result, it creates the initial 30K file, which never changes. I used verbose mode, but didn’t give me much info. Listed all the qubes. Asked “Do you want to proceed?” I said “y”, then I entered password, it gave me this:

 2022-01-012 10:25:46,974 [MainProcess selector_events.__init__:59] asyncio: Using selector: EpollSelector 
 Making a backup ... 0.00%

After a half hour i hit ctl-c and it gave me:

  Backup error: Backup cancelled
7

Well, at this point I’d try create a backup in vault (and later would copy it anyway from there to the other disk).

8

Not sure why you think vault would work better than other qubes? Trying it now, but so far same results, just sitting at 30K.

9

Apologize, forget my previous message. English is not my first language. I guess if you’d wrote “I’ve tried doing it to different qubes (meaning to primary disk as well)”, I’d probably understood properly.

I guess it doesn’t matter if you try to back up some small qube only, like sys-net?

10

Just to be clear: I meant to create a backup in a vault, not to a secondary disk attached to vault.
Also, did you start the qube you are trying to backup to? Did you try to exclude it from backup?

11

Yes, I tried to do a backup of just a few small qubes in the home directory of vault. Vault is running. Vault was not included in the backup. Same results.

12

Coming back to this in hopes of not having to reinstall everything.

A factor that determines when the backup freezes seems to be if any of the qubes I’m backing up are running. When some are running, it creates the initial 30K file and never gets past that point. Is there something in 4.1 that requires qubes to be halted during backup?

When I do shut them all down I can get about 10% into the backup before it freezes (which is 9.5GB for the current test), then the same thing happens, progress ceases.

13

Either something changed or my assertion that the backup freezes when all qubes are shutdown was mistaken. Here are my current observations:

  • I have now been able to get every qube (of the ones I was trying before) to backup successfully individually while it is shut down

  • I have also successfully backed up groups of qubes now (if they are all not running)

  • If I try to backup a collection of qubes and some are running that seems to be when I can get it to run for 6-9GB before freezing. Other times it freezes at 30K, guessing it depends which qube it starts with and whether it is running.

  • If I try to backup an individual qube that is running it doesn’t get past 30K, in other words it freezes immediately

/u/Xenepa on reddit suggested there might be something wrong with the snapshots Qubes uses for backup when an individual qube is running.

Very happy I can backup now without reinstalling. But having to shutdown qubes for many hours everytime I run a backup is still far from ideal and seems to be unique to me. Let me know if there’s anything you want me to check to investigate what is going on.

Note: This was a fresh install of 4.1rc3, now updated to 4.1rc4 with lots of qubes restored from a backup made in 4.0.4

14

Well, at this point I’d certainly backup my data, would do a clean install of rc4, would do a test backup then, and then would restore data from rc3, not from 4.0.4.

15

Just updating to say that I’m now using the latest (officially released) 4.1 and this has not fixed itself. I will probably try reinstalling as @enmus suggests as soon as I can find the time. But posting this in case anyone has other ideas?

thanks.

16

I reinstalled a fresh Qubes 4.1 and am getting the same behavior.

Just tried to run a backup of vault while it is running and it is hanging at 30K. This is deeply disappointing, and I have not found anyone else having this problem in the months that I’ve been experiencing it.

To make matters worse, in my 2 day project of shutting down and backing up all my qubes I ran into this problem: Restore failure - Unexpected EOF - retrieved data backup size exceed expected size · Issue #7198 · QubesOS/qubes-issues · GitHub

I am not using btrfs as others in that thread seem to be. I did find that compressing my backups seemed to greatly reduce if not eliminate that problem. I had been doing uncompressed to try to save time as suggested by others, because this has all been very time consuming .

Back to my original problem I decided to create a fresh qube and try backing that up to see if there was something about my qubes created in 4.0 that was not working in 4.1. This did not help, still hangs at 30K if qube is running during backup. So it’s really strange that no one else has experienced this.

17
18

What do you think about testing if your drive has failed?

19

If I stop the qube I can backup and restore no problem. I just completed a full backup of my system and reinstall from this same drive(dozens of qubes). As stated above I’ve replicated this behavior on multiple drives.

20

Yes, I meant to check the drive to read from, and not to where to.
Did you try backintime, RepRap, snapper, pigz as compression… (I haven’t so I’m not sure if it can actually backup qubes, or only dom0)?

21

[quote=“enmus, post:20, topic:8660, full:true”]
Yes, I meant to check the drive to read from, and not to where to.[/quote]

You mean the drive it’s installed on? I haven’t tested that. It is a RAID1 - so 2 redundant drives.

No, never used those.

22

Is there any output on

sudo mdadm /dev/md0?

I’d do if I were you. I tried it in a template for other reasons (preparing restoring strategy). Nothing to loose

23

no /dev/md0, there is /dev/md126, which is 2 devices for the storage
and /dev/md127 is 976MB, which is 2 devices and has the boot info on it

24

Re-installing with BTRFS seems to have fixed this problem per:

However, now the system is constantly overloaded, so not really viable solution. Gonna start a new thread on it.

1 Like
25

Great, thanks!