LVM physical volume (pv) for vg qubes_dom0 not found after installing on a USB drive

User Support General
1

I have written the Qubes installer onto a USB drive with dd, and then proceeded through the installation. The installation destination was set to another USB drive that was completely empty (without any partitions or even a GPT on it), and all of the settings were left at the defaults. The installation proceeded without any errors, but after installing the OS it wouldn’t boot properly.

After the computer restarts, I can boot into GRUB normally, and the encryption password for the LUKS partition is asked. However after that, it hangs for a few minutes before reporting that the lvm stuff has failed.

Warning: /dev/mapper/qubes_dom0-root does not exist
Warning: /dev/qubes_dom0/root does not exist
Warning: /dev/qubes_dom0/swap does not exist
Warning: crypto LUKS UUID 12735180-453f-4e94-bc38-6a48d1265dd not found

I saw these interesting lines in journalctl:

Scanning devices dm-0  for LVM logical volumes qubes_dom0/root
    Warning: File locking is disabled
    Warning: Couldn't find device with uuid rvmLUf-Eyy5-Bfdc-0szu-lcp2-QHuG-ffdtGe
    Warning: VG qubes_dom0 is missing PV rvmLUf-Eyy5-Bfdc-0szu-lcp2-QHuG-ffdtGe (last written to /dev/mapper/luks-12735180-453f-4e94-bc38-6a48d12656dd).

The lines after that explained that activating the logical volumes for root and swap will be refused, and that ‘–activatemode partial’ is needed to make it continue anyways.

Attempted Solutions
I corrected the hints to the search command in the GRUB boot option by changing them from ‘hd1’ and ‘ahci1’ to ‘hd0’ and ‘ahci0’. The OS loaded Linux and xen much faster, but the error still occurred.
I hypothesize that is because the search command uses the UUID of the partition, so even with incorrect hints it would still set root to the corect partition, but would take longer to do so. (so what I did didn’t fix anything)

I have tried copying the files in the ESP (EFI System Partition) /efi/qubes directory to efi/boot as suggested here, along with using efibootmgr to remove the ‘qubes’ boot entry and instead use the ‘USB drive’ boot entry (which should use the /EFI/boot/bootx64.efi GRUB), but neither of those fixed it, likely because it’s not a problem with booting GRUB.

I have also searched this in a search engine, and all of the other results I found related to LVM thin provisioning and file locking, which I think don’t apply to this problem.

Thank you in advance for any help, and I’m willing to give any more information if it’s needed. Here is my rdsosreport log from initramfs.

2

Boot from some other OS (e.g. from Qubes OS installer or some Live Linux OS) and try to mount your dom0 there for a test.
Then check the UUIDs of the disk/partitions.
Check if you’ll see the PV rvmLUf-Eyy5-Bfdc-0szu-lcp2-QHuG-ffdtGe there.

3

On second thought, maybe you have Qubes OS installed on another disk without disk encryption and then with this disk connected you’ve installed Qubes OS on the USB disk without encryption as well?
In that case it could be confused because you have the same VG qubes_dom0 available on two different disks (PV).

4

Hi. Sorry for taking a long time to reply, as I had other stuff to do. I tried booting Debian (which is installed in the internal drive of my laptop) and decrypted the qubes LUKS partition with

# cryptsetup luksOpen /dev/sda3 qubes_dom0

and using

$ vgdisplay

showed that the PV starting with rvm was still not found.

About your second suggestion, there was only 1 installed copy of Qubes OS on that computer at the time, and to my knowledge even if there were 2 identically named VGs, LVM would be able to differentiate between them by using UUIDs. (I’ve done this before when I upgraded the internal drive on my laptop from a 256 GB drive to a 1 TB one on which I created a new installation with the same vg name. Later when I wanted to access data on the 256 GB drive, I used a USB to PCIe bridge to connect it externally, and used vgrename to fix the name conflicts)

Thank you for your help and for responding. I hope you have a good day.

5

In rdsosreport.txt you can see:

+ command -v lvm
+ lvm pvdisplay
  WARNING: Couldn't find device with uuid rvmLUf-Eyy5-Bfdc-0szv-Icp2-QHuG-ffdtGe.
  WARNING: VG qubes_dom0 is missing PV rvmLUf-Eyy5-Bfdc-0szv-Icp2-QHuG-ffdtGe (last written to /dev/mapper/luks-12735180-453f-4e94-bc38-6a48d12656dd).
  --- Physical volume ---
  PV Name               [unknown]
  VG Name               qubes_dom0
  PV Size               362.97 GiB / not usable 2.00 MiB
  Allocatable           yes 
  PE Size               4.00 MiB
  Total PE              92920
  Free PE               86784
  Allocated PE          6136
  PV UUID               rvmLUf-Eyy5-Bfdc-0szv-Icp2-QHuG-ffdtGe
   
  --- Physical volume ---
  PV Name               /dev/mapper/luks-ec2a0e21-2042-438e-82d5-ace7ba0b85bd
  VG Name               qubes_dom0
  PV Size               114.96 GiB / not usable 2.00 MiB
  Allocatable           yes 
  PE Size               4.00 MiB
  Total PE              29430
  Free PE               29420
  Allocated PE          10
  PV UUID               41LMMf-ICNN-i9nE-gfMT-qwua-PVCW-Rjdnl3

So it seems like two PVs are used for the qubes_dom0 VG.
Remove the missing PV and try to boot.

6

I did that with

vgreduce --removemissing --force qubes_dom0
  WARNING: Couldn't find device with uuid rvmLUf-Eyy5-Bfdc-0szv-Icp2-QHuG-ffdtGe.
  WARNING: VG qubes_dom0 is missing PV rvmLUf-Eyy5-Bfdc-0szv-Icp2-QHuG-ffdtGe (last written to [unknown]).
  WARNING: Couldn't find device with uuid rvmLUf-Eyy5-Bfdc-0szv-Icp2-QHuG-ffdtGe.
  WARNING: Removing partial LV qubes_dom0/root-pool.
  WARNING: Couldn't find device with uuid rvmLUf-Eyy5-Bfdc-0szv-Icp2-QHuG-ffdtGe.
  logical volume "root" successfully removed.
  WARNING: Couldn't find device with uuid rvmLUf-Eyy5-Bfdc-0szv-Icp2-QHuG-ffdtGe.
  WARNING: volume "root-pool" successfully removed.
  WARNING: Removing partial LV qubes_dom0/swap.
  WARNING: Couldn't find device with uuid rvmLUf-Eyy5-Bfdc-0szv-Icp2-QHuG-ffdtGe.
  Logical volume "swap" successfully removed.
  Wrote out consistent volume group qubes_dom0.

and I had to use --force, because without it I would get the errors

  WARNING: Couldn't find device with uuid rvmLUf-Eyy5-Bfdc-0szv-Icp2-QHuG-ffdtGe.
  WARNING: VG qubes_dom0 is missing PV rvmLUf-Eyy5-Bfdc-0szv-Icp2-QHuG-ffdtGe (last written to /dev/mapper/luks-12735180-453f-4e94-bc38-6a48d1265dd).
  WARNING: Couldn't find device with uuid rvmLUf-Eyy5-Bfdc-0szv-Icp2-QHuG-ffdtGe.
  WARNING: Partial LV root-pool needs to be repaired or removed.
  WARNING: Partial LV root to be repaired or removed.
  WARNING: Partial LV swap needs to be repaired or removed.
  WARNING: Partial LV lvol0_pmspare needs to be repaired or removed.
  There are still partial LVs in VG qubes_dom0.
  To remove them unconditionally use: vgreduce --removemissing --force.
  To remove them unconditionally from mirror LVs use: vgreduce --removemissing --mirrorsonly --force.
  Warning: Proceeding to remove empty missing PVs.
  Warning: Couldn't find device with uuid rvmLUf-Eyy5-Bfdc-0szv-Icp2-QHuG-ffdtGe.

I’m pretty sure that it removed all of the LVs, as lvdisplay didn’t show any LVs. Booting it failed, with the message

[ TIME ] Timed out waiting for device dev-disk-by\x2duuid-12735180\x2d453f\x2d4e
[DEPEND] Dependency failed for systemd-cryptsetup@luks\x2d12735180\x2d453f\x2d4e
[DEPEND] Dependency failed for cryptsetup.target - Local Encrypted Volumes.
[ TIME ] Timed out waiting for device dev-disk-by\x2duuid-ec2a0e21\x2d2042\x2d43
[DEPEND] Dependency failed for systemd-cryptsetup@luks\x2dec2a0e21\x2d2042\x2d43
Warning: /dev/mapper/qubes_dom0-root does not exist
Warning: /dev/qubes_dom0/root does not exist
Warning: /dev/qubes_dom0/swap does not exist

I have the rdsosreport.txt for that, but I don’t think it’s needed as it seems pretty clear that there were no LVs in the VG so it failed to start. I’ll reinstall it and see if that fixes it. Thank you for the help.

7

Try to reinstall the Qubes OS on your USB disk without any disks containing the installed unencrypted Qubes OS connected.

8

I tried reinstalling Qubes OS from the installer. For some reason it took a very long time (~9 hours) and throughout that time the USB drive was hot, so a lot of data was being transferred.

The disks connected during the installation were:

/dev/nvme0n1 # the internal disk with Debian (encrypted, LUKS)
/dev/sda # the disk with the Qubes OS installer, written to it by dd
/dev/sdb # the destination disk, where I installed Qubes OS at

To clear any confusion, there were no other unencrypted or encrypted Qubes OS disks connected to the computer. I had one installation like this before, but it had encryption with LUKS, and was physically disconnected from the computer throughout this whole ordeal.
Besides, even if there was another installation, I doubt that it could have any affect, as LVM uses UUIDs for PVs, so Qubes OS would be able to differentiate them.

When booting the newly installed Qubes OS on the USB disk, it encountered the same problem as earlier of timing out waiting for the disk, with the only difference being the UUID of the disk, as UUIDs are random.

Sorry for taking so long to respond and thank you for the help so far; I’ll add the rdsosreport when I get home incase it’s needed.

9

The new rdsosreport is here.

I also tried accessing the root LV. I decrypted LUKS with

# cryptsetup luksOpen /dev/sda3 decrypted_qubes_dom0

vgdisplay and lvdisplay showed the qubes_dom0 VG and root-pool, root, and swap LVs respectively normally. I was able to normally mount /dev/qubes_dom0/root and inside it there were the normal qubes OS file hierarchy. I enctounered no errors throughout this process, which confuses me as to why it fails to start. Thank you for the help so far.