Qubes backup important question

I recently managed to upgrade to R4.2 but I was met with something that I really did not expect.

I first tried upgrading directly from 4.1 but had some errors/complications probably because of the tweaks I did during the years.

So I just said cool I’ll backup and restore once I install it fresh again…
Backed up all the most important stuff (password and wallets VMs) but when I restored everything just disappeared, I don’t even know what’s the point in having backups if this happens? Is this some kind of error? am I not looking in the right places in the restored VMs?

P.S. fortunately I had another machine with everything even if it was a little dated, managed to lose 50 bucks still tho ;_; lesson learned

Could you give us more details? Did you make the backup and restore using the command line (which commands exactly?) or UI? Did you verify your backup? Did you get any error messages? Or just empty restored VMs? Or no restored VMs?

3 Likes

I have the same questions as @fsflover

In addition, if you restore qubes without their template, they won’t have any programs in their menu entry IIRC :thinking: (but they should at least appear)

if the restore program took a few minutes for restore and it wasn’t instant, this mean it did something

2 Likes

Your data is almost certainly still safely stored in the backup. It sounds like some kind of error (possibly user error) when attempting to restore from the backup, but it should certainly be solvable if you provide some more information. If needed, you can always use the emergency restore procedure to recover all of your data from the backup, but even that probably won’t be necessary.

2 Likes

@fsflover @solene @adw thanks everybody for the response! This was my procedure:

I backed up my most important VMs (which were my password, wallet and workstation) with the whonix template included in the backup.
My backup device is a USB which I completely formatted to ext4 before I put my backup there.

After this I went and used another USB device to reinstall qubes r4.2 completely fresh on my machine.
Installation went successfully and then I proceeded to restore my backup using the qubes GUI option.

the restoring process seemed to be going smooth and finished with no errors but my VMs seem to be completely wiped out of their respective homes and files with some exceptions: for example, in my wallet VM there still were all my /opt installed wallets (the software) despite being completely wiped out of the actual wallet files.
password VM was completely wiped instead with no sign of my keepass vault or my pgp keys in sight.

@fsflover I just tried verifying my backup as soon as I turned on my machine now and this is what it gives me:
→ Verifying whonix-ws-16…
→ Verifying vault…
→ Verifying wallets…
Extracting data: 10.6 GiB to restore
Finished with errors!
failed to decrypt /home/[USER]/QubesIncoming/backup#restore-pi5hf5vc/vm11/private.img.009.enc: b’scrypt: Input is not valid scrypt-encrypted block\n’
Partially restored files left in /var/tmp/restore_*, investigate them and/or clean them up
Please unmount your backup volume and cancel the file selection dialog.include

I tried checking the backups computer files and there seems to be another folder named /home.orig which I thought would store all the backup files but they seem to be empty too.

thanks for the help I hope we could figure this out!

This is because /opt is on a different Qubes OS storage volume (the ‘root’ volume), which in case of an AppVM comes straight from its TemplateVM.

If you limit the VM selection for restore to just e.g. the “wallets” VM and try that let’s say 5 times, does it always fail at exactly the same e.g. .009.enc chunk? Then that chunk is likely to be corrupted. You might still be able to pull out enough data from the other uncorrupted chunks if you’re lucky:

However if the error sometimes occurs at a different chunk, then the data itself might be fine and it could be due to something like overheating, bad RAM, or maybe a variation on this recent incompatiblity with some AMD systems.

3 Likes

If it’s an usb memory stick, it may be corrupted (theses things are a bit better than floppy but not very reliable). In the qube used to mount the USB devices, you could check dmesg output to see if there are errors related to ext / I/O / disk etc…

seems to me that most of the backups are corrupted then.

I’m keeping a little log and so far I tried restoring wallets alone for a couple times and it just repeats the same message:

Finished with errors!
failed to decrypt /home/[USER]/QubesIncoming/backup#restore-f0680th6/vm20/private.img.000.enc: b’scrypt: Input is not valid scrypt-encrypted block\n’
Partially restored files left in /var/tmp/restore_*, investigate them and/or clean them up

This is instead the dialog box when I try and restore Vault:

Finished with errors!
failed to decrypt /home/highcee77/QubesIncoming/backup#restore-4uw96twk/vm11/private.img.009.enc: b’scrypt: Input is not valid scrypt-encrypted block\n’
Partially restored files left in /var/tmp/restore_*, investigate them and/or clean them up

I’ll try restoring all the other backups and these two for some other times and will let you know, thank you

edit: I doubt it’s gotta do with AMD incompatibility cause all my machines run intel (even though this r4.2 one has “new gen” intel which seems to have better compatibility with r4.2 but a lot of things did not work in 4.1 now that I think about it, r4.0 was out of question: full incompatibility)

2nd edit: I just ran through restoring backups a couple of times and out of all my VMs that I backed up the only 2 ones giving me trouble are wallets and vault ;_; their respective errors are in up in this post.
to be fair the other VMs that I backed up were real tiny system VMs and both whonix template.
dom0 backup doesn’t give trouble either

1 Like

It might be but after I noticed the backups weren’t working I then used the same USB to copy all my old passwords and files from my other old machine and they all just copied and pasted just fine

UPDATE

I spent all afternoon playing with this and for some reason my wallet and vault VMs are corrupted for no reason when all the other VMs I backed up are fine with no errors.

What I tried to do was to use the emergency restore procedure as @adw said and when I get to step 6 in the procedure there seems to be some kind of error when building the private.img of both VMs.
this is what appears on terminal for both at the 6th step:

scrypt: Input is not valid scrypt-encrypted block

gzip: stdin: unexpected end of file
tar: Unexpected EOF in archive
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now

in response to this command:

find vm[VAULT/WALLETS] -name ‘private.img.*.enc’ | sort -V | while read f_enc; do f_dec=${f_enc%.enc}; echo “$backup_id!$f_dec!$backup_pass” | scrypt dec -P $f_enc || break; done | gzip -d | tar -xv

@rustybird you are my savior! that thread you posted is a literal goldmine for everyone stuck with a problematic/corrupted backup!! I really can’t thank you enough I managed to find my most recent keepass vault with all of my passwords and seeds so I can restore my wallets too!

I can say I lost some data in the backup but I’m so happy I managed to recover the most important things of all thank to you!
I wish I could’ve seen my face as soon as I noticed all the .kbdx files bgrep found in the .img xD

Ha, nice! It’s so lucky that password manager database files are pretty small, otherwise they would be a whole lot harder to recover. *shudder*

So you deleted a .enc file that was corrupted and preventing the backup to continue further? That’s great if it worked! That’s cool to know that even if a backup is partially corrupted it’s possible to extract as much as possible from it.

This is why it’s really important to verify (i.e., test restore) your backup right after creating it. If the verification fails, discard that backup and create another one until the verification succeeds. Until you have a backup that passes verification, your data isn’t really backed up. As the saying goes, “Backups always succeed. It’s restores that fail.”

Related issue:

2 Likes

In that case, the backup may have been corrupted when moved to the USB media disk if it was made locally first. Testing it first wouldn’t have helped.

But yeah, it’s always important to verify the backup before using it.

In the Qubes Backup Restore tool there is a checkbox option to just test without extracting anything :+1:

But you should also keep more than one copy of the backup in different places. The claim isn’t that verification is by itself sufficient to prevent data loss; following other best practices is also necessary.

Yes, this is what I was referring to. Thank you for making it explicit. You can also use it on the command line with --verify-only.

1 Like

unfortunately too many blocks were corrupted to have the whole backup working again, if I remember well gzrecover got around 1/2GB of data out of the 6GB backup. it was expected tho.

It went well on single VM images tho. once I figured out how to check I saw that it was just around 6 blocks that were corrupted out of a total of around 40 in my vault VM, which meant it was highly possible that bgrep could find my kdbx considering it is just 20kB!

I did lose all my other data tho, fortunately it wan’t really important. I gotta say that I didn’t really try to recover it tho since I don’t really know how you could do that. But if you are looking for something really specific there’s a probability that it’s hiding in the midst of all the good data that gzrt found

1 Like

you are very much right! I can really say I’ve learnt my lesson these days xD. I thought I was gonna have an heart attack when I saw all my VMs completely wiped

You should have regular backups, not just one when migrated, in addition there is a simple rule named 3-2-1, this is the minimum for safe backups.

  • 3 copies of your data
  • on 2 different medium (hard drive, usb mem stick, cloud)
  • with 1 backup on an extra site
3 Likes