All vms suddenly stopped working (cant start any vm, even sys-net)

User Support General
1

qubes was working as expected but one day i logged in, and for some resson none of the vms started

if i try something like qvm-start personal heres what i get:

user@dom0:~/Desktop/qvm-start personal$
Traceback (most recent call last):
File “/usr/lib/python3.13/site-packages/qubesadmin/app.py”, line 361, in qubesd_call
sock.recv(4)
FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/bin/qvm-start”, line 5, in
sys.exit(main())
File “/usr/lib/python3.13/site-packages/qubesadmin/tools/qvm_start.py”, line 175, in main
args = parser.parse_args(args, app=app)
File “/usr/lib/python3.13/site-packages/qubesadmin/init.py”, line 431, in parse_args
action.parse_qubes_app(args)
File “/usr/lib/python3.13/site-packages/qubesadmin/init.py”, line 192, in parse_args_app
namespace.domains = (app.domains,)
File “/usr/lib/python3.13/site-packages/qubesadmin/app.py”, line 107, in getattr
if not self.app.blind_mode and attr not in self.dict:
File “/usr/lib/python3.13/site-packages/qubesadmin/app.py”, line 143, in contains
self.refresh_cache()
File “/usr/lib/python3.13/site-packages/qubesadmin/app.py”, line 73, in refresh_cache
vm_list_data = self.app.qubesd_call(“dom0”, “vm-list”)
File “/usr/lib/python3.13/site-packages/qubesadmin/app.py”, line 863, in qubesd_call
raise qubesadmin.exc.QubesDaemonCommunicationError(
qubesadmin.exc.QubesDaemonCommunicationError: Failed to connect to qubesd service: [Errno 2] No such file or directory

user@dom0:~/Desktop$

and if i try to start sys-net:
Error: Check of pool qubes_dom0/vm-pool failed (status:64). Manual repair required!
Aborting. Failed to locally activate thin pool qubes_dom0/vm-pool.

1 Like
2

What output do you get for

lvscan

In dom0 terminal

3

All are inactive except ‘/dev/qubes_dom0/root-pool’, ‘/dev/qubes_dom0/root’, and ‘‘/dev/qubes_dom0/swap’.

4

@corny

5

Can it be you exhausted all free disk space with your qubes?

It seems that LVM is falling apart.

1 Like
6

I’m confident that’s not the issue. @balko

7

OK. Because I had a similar situation when I exhausted all free space in older version of Qubes OS (R3.* or R4.0).

8

Please help! I’ve lost access to all my data!

9

Your LVM metadata is borked, the Error 64 is referring to manually repairing your metadata with lvconvert --repair

In dom0 terminal, de-activate your logical volumes;
lvchange -an qubes_dom0

Repair your pool metadata;
lvconvert --repair qubes_dom0/vm-pool

If you get transaction ID errors, there are additional steps. If not, continue here.

Reactivate logical volumes;
lvchange -ay qubes_dom0

You can now mount your private volumes to manually backup data, or just reboot.

2 Likes
10

I had similar error codes. Ended up doing a fresh re install. However i think the problem with mine (still investigating) was my sys-net had a pci device which did not exist.

R 4.3 renamed my Ethernet pci device. Whenever i tried to start sys-net in got similar error codes. On my new install of 4.3, I made sure to disable the old pci device (5c:00.0?******* unkown vendor) which does not exist anymore.

Try adding ‘qubes.skip_autostart’ to GRUB on boot. See if it persists

11

Transaction id 21472 from pool “qubes_dom0/vm-pool” does not match repaired transaction id 21471 from /dev/lvol0_pmspare.

12

Issue persists unfortunately.

13

Ok, so you need to manually edit your volume group config as well;

De-activate LV’s;
lvchange -an qubes_dom0

Backup metadata;
vgcfgbackup qubes_dom0 -f /etc/lvm/backup/qubes_dom0

Edit your backup config file - search for transaction ID number and replace with correct ID, eg. change 21472 to 21471 and save;
sudo nano /etc/lvm/backup/qubes_dom0

Restore edited metadata;
vgcfgrestore --force qubes_dom0 -f /etc/lvm/backup/qubes_dom0

Reactivate LV’s;
lvchange -ay qubes_dom0

Use lvscan to check if LV’s are now all ACTIVE. If they are, reboot and qubesd should load normally.

1 Like
14

user@dom0:~/Desktop$ lvchange -an qubes_dom0
WARNING: Running as a non-root user. Functionality may be unavailable.
/dev/mapper/control: open failed: Permission denied
Failure to communicate with kernel device-mapper driver.
Incompatible libdevmapper 1.02.199 (2024-07-12) and kernel driver (unknown version).
/run/lock/lvm/V_qubes_dom0:aux: open failed: Permission denied
Can’t get lock for qubes_dom0.
Cannot process volume group qubes_dom0

user@dom0:~/Desktop$ sudo lvchange -an qubes_dom0
Logical volume qubes_dom0/root contains a filesystem in use.
Logical volume qubes_dom0/swap in use.
Device qubes_dom0-root-pool_tmeta (253:1) is used by another device.
Device qubes_dom0-root-pool_tdata (253:2) is used by another device.

user@dom0:~/Desktop$ vgcfgbackup qubes_dom0 -f /etc/lvm/backup/qubes_dom0
WARNING: Running as a non-root user. Functionality may be unavailable.
/dev/mapper/control: open failed: Permission denied
Failure to communicate with kernel device-mapper driver.
Incompatible libdevmapper 1.02.199 (2024-07-12) and kernel driver (unknown version).
/run/lock/lvm/V_qubes_dom0:aux: open failed: Permission denied
Can’t get lock for qubes_dom0.
Cannot process volume group qubes_dom0

user@dom0:~/Desktop$ sudo vgcfgbackup qubes_dom0 -f /etc/lvm/backup/qubes_dom0
Volume group “qubes_dom0” successfully backed up.

user@dom0:~/Desktop$ sudo nano /etc/lvm/backup/qubes_dom0

user@dom0:~/Desktop$ vgcfgrestore --force qubes_dom0 -f /etc/lvm/backup/qubes_dom0
WARNING: Running as a non-root user. Functionality may be unavailable.
/dev/mapper/control: open failed: Permission denied
Failure to communicate with kernel device-mapper driver.
Incompatible libdevmapper 1.02.199 (2024-07-12) and kernel driver (unknown version).
WARNING: Failed to check for active volumes in volume group “qubes_dom0”.
/run/lock/lvm/P_global:aux: open failed: Permission denied

user@dom0:~/Desktop$ sudo vgcfgrestore --force qubes_dom0 -f /etc/lvm/backup/qubes_dom0
Volume group qubes_dom0 has active volume: root.
Volume group qubes_dom0 has active volume: root-pool.
Volume group qubes_dom0 has active volume: root-pool_tdata.
Volume group qubes_dom0 has active volume: root-pool_tmeta.
Volume group qubes_dom0 has active volume: swap.
WARNING: Found 5 active volume(s) in volume group “qubes_dom0”.
Restoring VG with active LVs, may cause mismatch with its metadata.
Do you really want to proceed with restore of volume group “qubes_dom0”, while 5 volume(s) are active? [y/n]: y
WARNING: Forced restore of Volume Group qubes_dom0 with thin volumes.
Restored volume group qubes_dom0.

user@dom0:~/Desktop$ sudo lvchange -ay qubes_dom0
Check of pool qubes_dom0/vm-pool failed (status:64). Manual repair required!

15

Did you actually find and edit the transaction ID number? Or just copy paste sudo nano /etc/lvm/backup/qubes_dom0 and do nothing?

1 Like
16

Found and edited.

17

Ok, try repair your volume group and pool metadata first, then run through the steps again.

Update VG metadata;
vgck --updatemetadata qubes_dom0

Update pool metadata;
lvconvert --repair qubes_dom0/vm-pool

Now follow my last steps again and see if you can activate LV’s.

2 Likes
18

Also show your output of sudo pvs, don’t post your luks drive UID though

1 Like
19

It now works. Thanks!

1 Like
20

This is happening more often in 4.3. In 4.2, it almost never happened. But now it’s happening after nearly every improper shutdown.

Someone experienced it here:

I then experienced it myself and posted a summary and the solution. After that, it happened to me two more times due to improper shutdowns.

This issue should be investigated to determine why it is occurring so frequently in 4.3. During the entire two years of 4.2, it never happened to me, and a similar issue occurred only once in 4.1.

3 Likes