No more Qubes available (except dom0) - How do I get them back?

I need help recovering my OS!

When I have fully started QubesOS, only dom0 is up and ready for use - but no Qube is booted up whatsoever.

When I then try to start a Qube manually from the Qube Manager, i.e. “sys-ubs”, then I get the error message:

“volume qubes_dom0/vm-fedora-29-root missing”

But I did certainly not delete anything from my system, so it cannot be missing. How can I get this up and running again?

(Btw, I got here because I had the following bug and ran the explained procedure: https://github.com/QubesOS/qubes-issues/issues/5160)

Thank you very much for any help!

1 Like

Take a close look at the output of ‘qvm-volume’ command. After that, you
may also want to look at ‘lvs’ to see if the volume exists.

It may be possible to do a ‘qvm-volume --revert’ to restore the template
to its former (hopefully working) state.

Thank you!

So “qvm-volume” gives me a whole list, but the column “REVERT_POSSIBLE” says “No” for every item. I guess that means that a “revert” is definitely impossible?

And when I use the “lvs” command, I do see the mentioned volume.

What are my next steps/options to get my data back?

1 Like

These are LVM volumes (LVM is the default pool type in Qubes). So I
suspect this is a Linux LVM storage issue; if so then there is a lot
more LVM expertise out on regular Linux forums.

What I can suggest you do now is back up the ‘*-private’ volumes of the
VMs you care about. They are basically just Linux block devices, so this
isn’t too difficult. A traditional way would be to mount an external
drive use ‘dd’.

You might even try using Qubes’ backup tool, although it might not work
since Qubes can’t access the volume to start the VM.

You could also try mounting the volume to see if its accessible. That’s
probably what I would do next.

1 Like

Thank you!

This issue is that when I do…

$ ls var/lib/qubes/appvms/some-important-qube

…it shows only 2 items: firewall.xml and icon.png -> /usr/share/icons/hicolor/… (and it’s like this for the entire list of Qubes)

So all the “private” images (= my data) seem to be “gone” - for an unknown reason. (The only thing that happened was my notebook dropping on the floor once.)

I suspect I need to use some forensic disk inspection tool on the hard drive, in order to try to find these “private image” files again… (If you have any other idea, it would be welcome.)

Please read the thread carefully.
In older versions of Qubes, data was stored in file backed images in
/var/lib/qubes/appvms/*
In Qubes 4.0 qube data is stored in lvm, and not in file images.(Well,
not by default)
You would not expect to see any other items in that location.

Do not look for “private image” files.
Look for LVM backing for the private images - you can then copy those off
the system, or mount them in dom0 and extract the data.
If the data is important to you then I suggest you image the disk before
doing anything else. Tools like clonezilla will help you to do this.
If the data isn’t that important, and not having backups suggests it
isn’t, then just go ahead and extract the data from the LVM images, as
tasket suggests.

[unman] unman https://forum.qubes-os.org/u/unman
August 29

Please read the thread carefully.
In older versions of Qubes, data was stored in file backed images in
/var/lib/qubes/appvms/*
In Qubes 4.0 qube data is stored in lvm, and /not/ in file images.(Well,
not by default)
You would not expect to see any other items in that location.

Do not look for “private image” files.
Look for LVM backing for the private images - you can then copy those off
the system, or mount them in dom0 and extract the data.
If the data is important to you then I suggest you image the disk before
doing anything else. Tools like clonezilla will help you to do this.
If the data isn’t that important, and not having backups suggests it
isn’t, then just go ahead and extract the data from the LVM images, as
tasket suggests.

Yeah, I should have clarified that LVM volumes are found under
‘/dev/qubes_dom0’ because that’s where the LVM volume group is linked.

But this suggests you’re somewhat out of your depth in handling errors
in Linux storage systems (or at least LVM). If you’re familiar enough
with basic shell commands I can suggest a way to backup the volumes,
such as:

sudo dd if=/dev/qubes_dom0/vm-personal-private | gzip

/mnt/externalhd/vm-personal.img.gz

That backs up the data for the ‘personal’ VM, for example.

Thank you!

When I inspect the folder you mention using…

$ ls -la /dev/qubes_dom0

…I get…

lrwxrwxrwx 1 root root 7 Aug 30 09:10 root -> ../dm-5
lrwxrwxrwx 1 root root 7 Aug 30 09:10 swap -> ../dm-6

(So there does not seem to be anything like vm-<my_template_name>-private in there, at least not directly visible.)

So then…

$ ls -la /dev/dm-5
$ ls -la /dev/dm-6

…returns…

brw-rw---- 1 root disk 253, 5 Aug 30 09:10 /dev/dm-5
brw-rw---- 1 root disk 253, 6 Aug 30 09:10 /dev/dm-6

$ ls -la /dev shows however that dm-0, dm-1, dm-2, dm-3, dm-4 are also available (not only dm-5 and dm-6).

So how do I extract the data from a dm-<X> “directly”?

Here’s the solution: Where is the Qubes' data stored?