Hi.
I have been using Qubes-OS for about 8 years. Love it!
About 2 years ago I bought a computer and restored 30+ VMs at a time flawlessly. Since upgrading to the latest version, I am unable to restore any VMs. I am not sure if the backups are failing to be created properly (they seem to finish without issue) or if the restore routine is not reading properly.
Upon restoring them (on a different machine) they either tell me the password is wrong or they fail with a bunch of red text and quickly as if the backup file had not even been read.
I am not new to Qubes and I have never posted here because I can usually figure out problem myself. For four months and over 200 tries, I have only had one backup restore successfully. I have been getting them up and running one by one by creating .tar.gz files of the home directory on each VM and then creating a VM on the target computer and extracting them. This problem has completely shut down my offices.
This has been going on for 4 months after both of my qubes computers had hard drive issues.
Observation: Dom0 has updated 4 times during these several months of complete hell. I thought maybe they were trying to fix this issue, but it continues. I donate to the development every year via bitcoin and I am having trouble talking myself into continuing that practice.
Anyone know what is happening? Anyone seen anything on this topic? (yes I have searched many times on this forum for answers over the last 4 months. nothing matches my issue.)
Any assistance would be appreciated.
I did that and there is nothing from 2024… The issue just started this year.
Did you verify the backup integrity after creating the backup?
Test restore your backup. Follow the restore procedure, selecting Verify backup integrity, do not restore the data. This step is optional but strongly recommended. A backup is useless if you can’t restore your data from it, and you can’t be sure that your backup is good until you try to restore.
Okay, I made a backup of one VM this morning and I just ran a “no restore” verify only on the same machine from which the backup was created and it finished successfully with no errors.
I am now doing real restore of that same backup to the same machine and will let you know if I get an error.
If no errors occur on the same machine that the backup was created on, I will take that backup file and try restoring it at my other location. The expected result will be a failure like the one you see above.
The same backup that we are talking about DID restore to the machine on which it was created. AND it is functional. It runs and has all its files. Now I am at 2 out of 200 attempts that worked.
I am now at my other location and attempting to verify the same file on the other machine. It fails go verify…
The file exists on a 245gb pendrive. It is where it was written by the backup utility. It was never copied or modified after being created by the backup utility.
For grins, I attempted to restore it on the machine here at the other location and this was the result - similar to the other failure I posted way at the top of this topic.
If you’ve backed up directly on this external drive then there is no need to verify the checksum. I thought you may have created backup on your system disk, verified it there and then copied it to the external disk and this copy could have an issue.
I have gone through 6 installations and 4 brand new external drives between the two machines. They were all installed from the same iso build to a pendrive. I am starting to think that the installation on the machine at the second location is messed up somehow. But the same thing happens when creating backups from location to for restore at the first location.
The only two successful restores in the last four months were - 1 an old backup from 2022 (second location a month ago) and - 2 the one I verified and restored on the same machine an hour or so ago (first location).
Can you verify this backup on both machines without restoring?
Maybe you have some custom modifications in dom0 of the second location machine in backup config that is not compatible with default backup config. E.g. something like this:
I appreciate all your thought and effort today. I have not made any mods to anything inside dom0 and both installations were done within the last six weeks. I cannot remember which ones I rebuilt via tar.gz or built from scratch. My old backups go back 8 years. I am not well versed in the system that makes up Qubes, but I am a power user. I have 115 machines showing in Qubes-manager - lots of them are failed restores. I only have 40 or so that I use plus their templates. If I ever get this working, I will have to go through them and find the ones that are just empty shells of the intended VMs they should have been and delete the dead ones.
The script you posted looks interesting but I have not worked in four months and I am trying to manage things from a windows laptop and the few VMs that do work. My next step will be to build a new machine to replace the oldest of the two and see what works and what doesn’t. Maybe someone will figure this out in the meantime.
I am really disgusted but I cannot have my crypto keys on anything but Qubes.
I will confirm that the version installed on both is the same and wait to see if anyone has any other ideas or experiences that jive with what is happening on mine.
Thanks for your time @apparatus let me know if there is anything else you can think of.
Maybe it’s a hardware problem? Like bad USB cable or you’re connecting your external drive to the second location machine using the USB hub and there is not enough power to supply the external disk.
This sounds like the problem I had late last year - I deduce it was caused by bad AMD microcode in a dom0 update Sept-Oct last year. Backup silently failing. An issue was originally opened in 2018, then closed. Re-opened 2023 totally different issue but symptoms similar (still open)… https://github.com/QubesOS/qubes-issues/issues/4493.
Been using the fake scrypt fix since then, though 4.2.1 did work OK last I looked. Odd thing was identical scrypt binary had no problems running in app VMs at the time.
I have been reading through some of these. It is nice to know that I am not the only one having this difficulty. Qubes is useless if each VM must be rebuilt from scratch with every hardware or Qubes version upgrade.
