No Qubes/VMs boot after latest updates

Already filed a bug report, but posting here for suggestions as my system is currently unuseable.

  1. applied latest updates yesterday
  2. rebooted today
  3. no VMs start

“libxenlight failed to create new-domain” pop ups for sys-net, sys-firewall, etc

/var/log/libvirt/libxl/libxl-driver.log shows:

libxl: libxl_dm.c:2857:stubdom_xswait_cb: Domain 1: Stubdom 2 for 1 startup: startup timed out
libxl: libxl_create.c:1975: domcreate_devmodel_started: Domain 1:device model did not start -9

Did you try to boot with old kernel in dom0 and old kernel in VMs?

Just tried 2 other kernels for sys-net, with no change.

I’m not sure about changing kernels in Qubes dom0, reading more about it.

Select the boot entry with old kernel in GRUB menu during Qubes OS boot.

I don’t see a Grub boot menu on this machine, I think it doesn’t use Grub.

I believe it uses Petitboot and Libreboot

I don’t know how are you booting, but you can use a hack and just rename the new kernel initramfs and vmlinuz files to some other names in /boot partition and copy the old kernel initramfs and vmlinuz files with the names of new kernel files.
But depending on how are you booting (heads or something maybe?) it could break the boot and you’ll need to rename the files back from some LiveCD or other system.

It’s a libreboot machine that I believe uses Petitboot instead of Grub

I don’t know much about Petitboot and Libreboot so I can’t help you here.

Aside from kernel you could also try to downgrade xen-hvm-stubdom-linux/xen-hvm-stubdom-linux-full packages to the ones that worked previously and maybe try to downgrade Xen if that won’t help.

Nevermind petitboot, I do get a Grub welcome screen actually, no kernel options. So I followed these instructions, but still not getting kernel options.

sudo nano /etc/default/grub
[update the following two lines, add if needed]
GRUB_DISABLE_SUBMENU=false
GRUB_SAVEDEFAULT=true
[save and exit nano]
sudo grub2-mkconfig -o /boot/grub2/grub.cfg

I see different options in the /boot folder, so may have to try your hack method.

You can also manually edit /boot/grub2/grub.cfg file so it’ll boot the kernel versions that you need instead of renaming the initramfs/vmlinuz files.

Did the name change hack.

uname -a
shows I’m in Linux 6.1.75 instead of 6.6.5

But sys-net still not booting.

Check the sys-net-dm.log, maybe there will be some errors there. And sys-net.log as well.
You can see then using Qube Manager or in dom0:

/var/log/xen/console/guest-sys-net.log
/var/log/xen/console/guest-sys-net-dm.log

/var/log/xen/console/guest-sys-net.log

hasn’t logged anything since failure to boot, except log file being opened

/var/log/xen/console/guest-sys-net-dm.log

has lots in it, but don’t see errors or warnings

I’d try to downgrade the xen-hvm-stubdom-linux/xen-hvm-stubdom-linux-full packages first to the versions that were previously installed and worked.
You can find the old working versions like this:
Get the History of an Installed Package with DNF | Thomas Stringer
If it won’t help then I’d try to downgrade Xen packages.

I think I’d be better to install Qubes OS on a separate disk and debug this issue there.

oh, can’t do this, no network

You can download the packages on another system and then move them to dom0 to install.
But it’s not really recommended from security standpoint.

I’ve got no sys-usb either though.