[qubes-users] Missing data after Qubes restore from backup

Hi,

I recently upgraded from R4.0 to R4.1 with a fresh installation.
I did make a backup of all my important Qubes to an external drive beforehand and everything completed successfully. I also checked "verify backup" -- no error reported.

On the new installation, I restored all my Qubes from the backup.
When starting to work with them again, I noticed that some data was apparently missing from some (not all!) of the Qubes. It turned out that in two of the restored VMs all data of the last six weeks (!) was completely missing. It looked like the VM had a state like from my last full VM backup using the backup utility (which was exactly that six weeks ago).

I tried restoring the VM again -- but same result. Searching for some issue like that only gave me [Issue #3498]. Apparently I was hit by the "flaw" that the verification doesn't detect missing data.

I guess I have no chance to restore my data whatsoever, since the old drive was wiped. Anyhow, just posting this: does someone has a clue *why* that could have happened? I don't want anyone to loose their work of six weeks, too.

[Issue #3498]: https://github.com/QubesOS/qubes-issues/issues/3498

Hi,

I recently upgraded from R4.0 to R4.1 with a fresh installation.
I did make a backup of all my important Qubes to an external drive beforehand and everything completed successfully. I also checked "verify backup" -- no error reported.

Do you often reboot your VMs? When Qubes creates a backup of a running VM it will do the backup using the VM state at startup. So if for example you started your VM (the ones with the outdated/missing data) 6 weeks ago, but since starting you added/edited/deleted data and you create a backup, without restarting the VM, the backup will not include those changes.

Do you often reboot your VMs? When Qubes creates a backup of a

running VM it will do the backup using the VM state at startup. So if for example you started your VM (the ones with the outdated/missing data) 6 weeks ago, but since starting you added/edited/deleted data and you create a backup, without restarting the VM, the backup will not include those changes.

Apparently, no. However, I made sure no VM was running (my host system crashed previously anyways). That means there's a difference between powering down a VM (e.g. qvm-shutdown) and having a powered down VM due to crash / power loss? Am I right that the backup tool uses the LV `vm-<name>-private` for backup only, but my data was in `vm-<name>-private-snap`?

Then, what about showing a warning that there exists a snap of a VM in the qubes-backup utility (at the same place where "The VM is running, backup will contain its state from before its start!" is shown as well), even if a VM is not running currently? That would have definitively saved me.

> Do you often reboot your VMs? When Qubes creates a backup of a running VM it will do the backup using the VM state at startup. So if for example you started your VM (the ones with the outdated/missing data) 6 weeks ago, but since starting you added/edited/deleted data and you create a backup, without restarting the VM, the backup will not include those changes.

Apparently, no. However, I made sure no VM was running (my host system crashed previously anyways). That means there's a difference between powering down a VM (e.g. qvm-shutdown) and having a powered down VM due to crash / power loss? Am I right that the backup tool uses the LV

Doing a gracefull shutdown or the VM crashing AFAIK is the same in terms of the backup function as we have discussed. If your VM crashed the state of the data on disk would still be the same compared to gracefully shutting down the VM.

Do you still have access to the old system to re-do the backup? Alternatively if you still have access to the old system you could manually mount the private volume of the VM and copy the data across.

Doing a gracefull shutdown or the VM crashing AFAIK is the same in terms of the backup function as we have discussed. If your VM crashed the state of the data on disk would still be the same compared to gracefully shutting down the VM.

My VM didn't crash -- the host system did. Therefore, I suspect the volume `vm-<name>-private` was never updated with the content of `vm-<name>-private-snap`, which probably occurs only on graceful VM shutdown (or on VM crash, but then the host system cleans things up and moves it correctly). If the host crashed, then this clean-up didn't take place. What a bad situation, indeed.

Do you still have access to the old system to re-do the backup? Alternatively if you still have access to the old system you could manually mount the private volume of the VM and copy the data across.

Sadly, I have no access to my old drive anymore as I pointed out earlier.

Crsi:

However, I made sure no VM was running (my host system
crashed previously anyways). That means there's a difference between
powering down a VM (e.g. qvm-shutdown) and having a powered down VM due to
crash / power loss? Am I right that the backup tool uses the LV
`vm-<name>-private` for backup only, but my data was in
`vm-<name>-private-snap`?

Spot on. That was the situation in R4.0.

Then, what about showing a warning that there exists a snap of a VM in the
qubes-backup utility (at the same place where "The VM is running, backup
will contain its state from before its start!" is shown as well), even if a
VM is not running currently? That would have definitively saved me.

I know it doesn't help you anymore, but R4.1 avoids the whole problem
by cleanly stopping crashed volumes when the computer is restarted:

Rusty

Just to clarify, the "verify only" option simulates a restore operation without actually writing any data from the backup to disk. This has two implications:

1. This option is not at all applicable when *creating* a backup (and should not even appear as an option there). [There's an open issue for a "create a backup, then verify it" feature, but it hasn't been implemented yet: https://github.com/QubesOS/qubes-issues/issues/1454]

2. Enabling this option when restoring will tell you that it was successful (if the simulation was successful), but no data on disk will change. This means, for example, that restoring from an older backup first, then restoring from a newer backup with "verify only" enabled, will result in the older VMs still being the ones on your disk.