None of the VMs working; qubes_dom0/vm-"VM-NAME"-"root OR private" is missing

dracoren · November 22, 2022, 2:05pm

Problem:
Qubes OS 4.1.1 on Acer Nitro 5 R9.
Disk password login works with external USB Keyboard, but at login wireless USB mouse/keyboard do not work.

After login, the QOS Devices applet fails to show USB devices section.
The qubes manager applet shows only dom-0 and qubes manager shortcuts.

When attempting to start any of sys-usb, sys-net, sys-whonix, or appvms the following messages are seen:

For AppVMs

volume qubes_dom0/vm-fedora-36-root is missing

volume qubes_dom0/vm-debian-11-root is missing

For sys-usb, sys-whonix etc

volume qubes_dom0/vm-dvm11-private is missing

Generally the message is:

volume qubes_dom0/vm-“VM-NAME”-“root OR private” is missing

Just before login prompt shows up, the following message can be seen in (after CTRL+ALT+F2 or Esc ) tty

A start job is running for LVM event activation on device 253:0

At login only USB wireless keyboard/mouse fail. The internal touchpad/keyboard function normally, allowing for login.

Just prior to this problem, I had updated dom0 and fedora36/debian11 templates on the previous day. Also, I had booted up with USB HDDs attached when this problem appeared.

I have checked some similar problems,

https://www.reddit.com/r/Qubes/comments/fszsu7/qubes_suddenly_broken_warning_devqubes_dom0root/

github.com/QubesOS/qubes-issues

Sum of all thin volume sizes exceeds limit. Critical Boot Issue

opened 04:45AM - 19 Nov 20 UTC

closed 04:51AM - 19 Nov 20 UTC

Jenn145

R: not applicable

### Qubes OS version: R4.0 ### Affected component(s) or functionality: Booting/Running the OS entirely --- ### Steps to reproduce the behavior: 1. Turn on computer, enter FDE login info 2. Dracut emergency prompt delivers error messages related to missing /dev/mapper/qubes_dom0-root 3. I use known working method for repair commented here by "notbyosmosis": https://github.com/QubesOS/qubes-issues/issues/5160 Instead of working as before, I receive the following errors following the second use of "lvconvert --repair qubes_dom0/pool00": Using default stripesize 64.00 KiB. WARNING: Sum of all thin volume sizes (1.10 TiB) exceeds the size of thin pools and the size of whole volume group (930.51 GiB)! For thin pool auto extension activation/thin_pool_autoextend_threshold should be below 100. Transaction id 4764 from pool "qubes_dom0/pool00" does not match repaired transaction id 4763 from /dev/mapper/qubes_dom0-lvol0_pmspare. WARNING: recovery of pools without pool metadata spare LV is not automated. WARNING: If everything works, remove qubes_dom0/pool00_meta1 volume. WARNING: Use pvmove command to move qubes_dom0/pool00_tmeta on the best fitting PV. ### Expected or desired behavior: Regular boot and ability to operate the OS ### Actual behavior: Dracut emergency prompt errors pertaining to this known issue: https://github.com/QubesOS/qubes-issues/issues/5372 https://github.com/QubesOS/qubes-issues/issues/5160 And a new issue following the use of a previous proven repair method, leading to new additional dracut emergency prompt errors. ### General notes: This issue is critical as one of my backups is corrupted, and the other is lost. I have been using Qubes for a little while now, and am dealing with a critical issue. After entering FDE login info, the boot-bar loads very slowly before entering an dracut emergency prompt and displaying: Warning: dracut-initqueue timeout - starting timeout scripts Warning: dracut-initqueue timeout - starting timeout scripts Warning: dracut-initqueue timeout - starting timeout scripts Warning: Could not boot. Warning: /dev/mapper/qubes_dom0-root does not exist Warning: /dev/qubes_dom0/root does not exist I have fixed this issue once in the past with the following steps: 1) At the Dracut emergency prompt, I ran: lvm vgscan vgchange -ay This informed me that there was an issue with activating qubes_dom0/pool00. At this point, I tried: lvconvert --repair qubes_dom0/pool00 I received the following error: Read-only locking type set. Write locks are prohibited. Can't get lock for qubes_dom0 cannot process volume group qubes_dom0 I exited back to the dracut prompt, and temporarily edited the /etc/lvm/lvm.conf file. I found the following line: locking_type=4 and changed it to: locking_type=1 2) I then retried the lvconvert command: lvm lvconvert --repair qubes_dom0/pool00 It worked! Recived a couple of warnings. I then re-ran the following: vgscan vgchange -ay I then exited back to the dracut prompt and edited the /etc/lvm/lvm.conf file, and changed the locking_type option back to what it was before: locking_type=1 changed it back to: locking_type=4 I then rebooted. QubesOS came up. For this second occurance of the issue, I attempted the previously used repair method, but received issues following step 2 , the errors displayed: lvm> lvconvert --repair qubes_dom0/pool00 Using default stripesize 64.00 KiB. WARNING: Sum of all thin volume sizes (1.10 TiB) exceeds the size of thin pools and the size of whole volume group (930.51 GiB)! For thin pool auto extension activation/thin_pool_autoextend_threshold should be below 100. Transaction id 4764 from pool "qubes_dom0/pool00" does not match repaired transaction id 4763 from /dev/mapper/qubes_dom0-lvol0_pmspare. WARNING: recovery of pools without pool metadata spare LV is not automated. WARNING: If everything works, remove qubes_dom0/pool00_meta1 volume. WARNING: Use pvmove command to move qubes_dom0/pool00_tmeta on the best fitting PV. lvm> (side note.. after completing the repair method the first time the error occured, an black icon/connection in sys-usb appeared and became present in every session. The connection option was named something like: "qubes_dom0/pool00_metadata") I have limited knowledge and experience with linux, however I can't imagine I have 1.10 TiB worth of VMs/data in this installation of Qubes. I think theres something up with the metadata pool that got split into a USB option, but I could easily be off. Please help me out if you can, I can't lose the progress I have made and am nervous messing around in the emergency prompt. --- ### I have consulted the following relevant [documentation](https://www.qubes-os.org/doc/): https://github.com/QubesOS/qubes-issues/issues/5160 https://github.com/QubesOS/qubes-issues/issues/5372 https://github.com/QubesOS/qubes-issues/issues/3810 https://github.com/QubesOS/qubes-issues/issues/3133 https://github.com/QubesOS/qubes-issues/issues/5372 ### I am aware of the following related, [non-duplicate](https://www.qubes-os.org/doc/reporting-bugs/#new-issues-should-not-be-duplicates-of-existing-issues) issues: https://github.com/QubesOS/qubes-issues/issues/5160 https://github.com/QubesOS/qubes-issues/issues/5372

lvconvert --repair looks like the way to go, but still not sure if any of these will work.

Anyone with the same problems? or suggestions?

Thanks.

dracoren · November 22, 2022, 5:25pm

One more finding.
I have a QOS411 updated to date (except for dom0 update), installed as a USB bootable, which has everything similar to the dual boot install as in OP, except that it doesn’t have a sys-usb qube.
It doesn’t have this issue.

dracoren · November 27, 2022, 5:11pm

Captured some images that might be useful.
The right lower output is from

“systemctl status lvm2-pvscan@253:0.service”

in dom0 terminal.