Hi.
I have been using Qubes-OS for about 8 years. Love it!
About 2 years ago I bought a computer and restored 30+ VMs at a time flawlessly. Since upgrading to the latest version, I am unable to restore any VMs. I am not sure if the backups are failing to be created properly (they seem to finish without issue) or if the restore routine is not reading properly.
Upon restoring them (on a different machine) they either tell me the password is wrong or they fail with a bunch of red text and quickly as if the backup file had not even been read.
I am not new to Qubes and I have never posted here because I can usually figure out problem myself. For four months and over 200 tries, I have only had one backup restore successfully. I have been getting them up and running one by one by creating .tar.gz files of the home directory on each VM and then creating a VM on the target computer and extracting them. This problem has completely shut down my offices.
This has been going on for 4 months after both of my qubes computers had hard drive issues.
Observation: Dom0 has updated 4 times during these several months of complete hell. I thought maybe they were trying to fix this issue, but it continues. I donate to the development every year via bitcoin and I am having trouble talking myself into continuing that practice.
Anyone know what is happening? Anyone seen anything on this topic? (yes I have searched many times on this forum for answers over the last 4 months. nothing matches my issue.)
Any assistance would be appreciated.
I did that and there is nothing from 2024⌠The issue just started this year.
Did you verify the backup integrity after creating the backup?
Test restore your backup. Follow the restore procedure, selecting Verify backup integrity, do not restore the data. This step is optional but strongly recommended. A backup is useless if you canât restore your data from it, and you canât be sure that your backup is good until you try to restore.
Okay, I made a backup of one VM this morning and I just ran a âno restoreâ verify only on the same machine from which the backup was created and it finished successfully with no errors.
I am now doing real restore of that same backup to the same machine and will let you know if I get an error.
If no errors occur on the same machine that the backup was created on, I will take that backup file and try restoring it at my other location. The expected result will be a failure like the one you see above.
The same backup that we are talking about DID restore to the machine on which it was created. AND it is functional. It runs and has all its files. Now I am at 2 out of 200 attempts that worked.
I am now at my other location and attempting to verify the same file on the other machine. It fails go verifyâŚ
The file exists on a 245gb pendrive. It is where it was written by the backup utility. It was never copied or modified after being created by the backup utility.
For grins, I attempted to restore it on the machine here at the other location and this was the result - similar to the other failure I posted way at the top of this topic.
If youâve backed up directly on this external drive then there is no need to verify the checksum. I thought you may have created backup on your system disk, verified it there and then copied it to the external disk and this copy could have an issue.
I have gone through 6 installations and 4 brand new external drives between the two machines. They were all installed from the same iso build to a pendrive. I am starting to think that the installation on the machine at the second location is messed up somehow. But the same thing happens when creating backups from location to for restore at the first location.
The only two successful restores in the last four months were - 1 an old backup from 2022 (second location a month ago) and - 2 the one I verified and restored on the same machine an hour or so ago (first location).
Can you verify this backup on both machines without restoring?
Maybe you have some custom modifications in dom0 of the second location machine in backup config that is not compatible with default backup config. E.g. something like this:
I appreciate all your thought and effort today. I have not made any mods to anything inside dom0 and both installations were done within the last six weeks. I cannot remember which ones I rebuilt via tar.gz or built from scratch. My old backups go back 8 years. I am not well versed in the system that makes up Qubes, but I am a power user. I have 115 machines showing in Qubes-manager - lots of them are failed restores. I only have 40 or so that I use plus their templates. If I ever get this working, I will have to go through them and find the ones that are just empty shells of the intended VMs they should have been and delete the dead ones.
The script you posted looks interesting but I have not worked in four months and I am trying to manage things from a windows laptop and the few VMs that do work. My next step will be to build a new machine to replace the oldest of the two and see what works and what doesnât. Maybe someone will figure this out in the meantime.
I am really disgusted but I cannot have my crypto keys on anything but Qubes.
I will confirm that the version installed on both is the same and wait to see if anyone has any other ideas or experiences that jive with what is happening on mine.
Thanks for your time @apparatus let me know if there is anything else you can think of.
Maybe itâs a hardware problem? Like bad USB cable or youâre connecting your external drive to the second location machine using the USB hub and there is not enough power to supply the external disk.
This sounds like the problem I had late last year - I deduce it was caused by bad AMD microcode in a dom0 update Sept-Oct last year. Backup silently failing. An issue was originally opened in 2018, then closed. Re-opened 2023 totally different issue but symptoms similar (still open)⌠https://github.com/QubesOS/qubes-issues/issues/4493.
Been using the fake scrypt fix since then, though 4.2.1 did work OK last I looked. Odd thing was identical scrypt binary had no problems running in app VMs at the time.
I have been reading through some of these. It is nice to know that I am not the only one having this difficulty. Qubes is useless if each VM must be rebuilt from scratch with every hardware or Qubes version upgrade.
The problem has been diagnosed and resolved by abandoning the two computers that both refused to backup/restore using the method under âQubes toolsâ. I purchased a new machine with two Xeon processors and 384 gigs of ddr4 (I got a deal on it). That was step one.
Before I bought it, I was able to spend several days at the store, in a back room, testing it to see if the backups made in the past would restore successfully. I had some success. So I took it home.
I believe the problem was something corrupt in the Qubes code (distinct on each failed machine) on both computers. The reason is explained in the rest of this post. Although the new box was able to restore old backups, it had several other issues when loaded up. So I had to diagnose that first, which included installing Qubes seven more times after bringing it home. None of the issues I had with it were backup/restore related, but this does prove that a clean ISO, burned without errors, is essential to making a stable machine that works. Likely, the other two machines had problems with the pendrive used to install the OS.
Once I got the new box running correctly, I started backing up the other two machines and restoring the VMs from each of them. The short answer is that the laptop (a gaming machine) backed up all its VMs and templates correctly every time but refused to to restore (even back to itself) claiming that the backup had a corrupt âxml headerâ or something to that effect. The new machine proves that claim to be false because they restored correctly.
The other failed computer had errors on the backup screen upon completion of some backups proving that it was incapable of backing up its VMs at times. That problem may be due to the fact that I was using a 4TB drive plugged in as a storage device. That storage device had previously been a failed installation of Qubes. I believe that the naming convention causes a problem on Qubes, being that the working installation sees another drive in-line that has the same name as the drive currently running Qubes. I will know if that is the case when I pull that extra drive and install Qubes again using all the same hardware, less the 4TB --and-- the recent (good) installation medium.
I have been using Qubes for about ten years. I am power user because its security, but I am not a power configuration expert. I suspect errors due to the name of the drives because when I had converted an old disk to USB storage one time, I remember odd errors in Qubesâ functionality while the drive was plugged in. No errors when plugged into a windows or debain machine.
