No Qubes/VMs boot after latest updates

I don’t know much about Petitboot and Libreboot so I can’t help you here.

Aside from kernel you could also try to downgrade xen-hvm-stubdom-linux/xen-hvm-stubdom-linux-full packages to the ones that worked previously and maybe try to downgrade Xen if that won’t help.

Nevermind petitboot, I do get a Grub welcome screen actually, no kernel options. So I followed these instructions, but still not getting kernel options.

sudo nano /etc/default/grub
[update the following two lines, add if needed]
GRUB_DISABLE_SUBMENU=false
GRUB_SAVEDEFAULT=true
[save and exit nano]
sudo grub2-mkconfig -o /boot/grub2/grub.cfg

I see different options in the /boot folder, so may have to try your hack method.

You can also manually edit /boot/grub2/grub.cfg file so it’ll boot the kernel versions that you need instead of renaming the initramfs/vmlinuz files.

Did the name change hack.

uname -a
shows I’m in Linux 6.1.75 instead of 6.6.5

But sys-net still not booting.

Check the sys-net-dm.log, maybe there will be some errors there. And sys-net.log as well.
You can see then using Qube Manager or in dom0:

/var/log/xen/console/guest-sys-net.log
/var/log/xen/console/guest-sys-net-dm.log

/var/log/xen/console/guest-sys-net.log

hasn’t logged anything since failure to boot, except log file being opened

/var/log/xen/console/guest-sys-net-dm.log

has lots in it, but don’t see errors or warnings

I’d try to downgrade the xen-hvm-stubdom-linux/xen-hvm-stubdom-linux-full packages first to the versions that were previously installed and worked.
You can find the old working versions like this:
Get the History of an Installed Package with DNF | Thomas Stringer
If it won’t help then I’d try to downgrade Xen packages.

I think I’d be better to install Qubes OS on a separate disk and debug this issue there.

oh, can’t do this, no network

You can download the packages on another system and then move them to dom0 to install.
But it’s not really recommended from security standpoint.

I’ve got no sys-usb either though.

Then you can boot from some LiveCD and mount the dom0 there to copy the files.
But it’s better to debug the issue on Qubes OS installed on another disk first.

Thanks for this, figured out what versions of packages were there before.

Are these qubes specific packages? Trying to get them in a Fedora LiveCD but they don’t exist. Where can I download them?

Yes, they are Qubes-specific.
You can download them from the repository:
Index of /r4.2/current/dom0/fc37/rpm/
You can check the signature of downloaded package like this:
Anyone got an AppVM or Debian standalone solution for Mullvad VPN? - #16 by unman

Tried:

  1. boot into Fedora live disk
  2. from here: Index of /r4.2/current/dom0/fc37/rpm/
    I downloaded: xen-hvm-stubdom-linux-4.2.9-1.fc37.x86_64.rpm AND xen-hvm-stubdom-linux-full-4.2.9-1.fc37.x86_64.rpm
  3. mount my Qubes OS encrypted harddrive
  4. copy downloaded packages to home directory
  5. reboot into Qubes
  6. ran
    rpm -K <each package name>
    to verify signatures
  7. ran
    sudo dnf localinstall <each package name>
  8. qvm-start sys-net

Got same results.

Tried downgrading xen, xen-hypervisor, xen-libs, xen-licenses and xen runtime to 4.17.3-4, but it gave me error:

The operation would result in removing the following protected packages: qubes-core-dom0

So not sure how to proceed.

I guess last thing to try is to downgrade the Xen packages:

[user@dom0 ~]$ dnf list installed | grep 4.17.3-5
python3-xen.x86_64                                2001:4.17.3-5.fc37                @qubes-dom0-cached
xen.x86_64                                        2001:4.17.3-5.fc37                @qubes-dom0-cached
xen-hypervisor.x86_64                             2001:4.17.3-5.fc37                @qubes-dom0-cached
xen-libs.x86_64                                   2001:4.17.3-5.fc37                @qubes-dom0-cached
xen-licenses.x86_64                               2001:4.17.3-5.fc37                @qubes-dom0-cached
xen-runtime.x86_64                                2001:4.17.3-5.fc37                @qubes-dom0-cached

But there could be some problems with dependencies which could require you to downgrade some other packages as well.
Tbh it’d be much easier if you could install test Qubes OS on a separate drive and update the packages there without updating just Xen packages (install a specific Xen version that worked for you and exclude the Xen packages from the updates).

Yes, qubes-core-dom0 (see above)

Well, that’d be a pain to go through all the package dependencies and downgrade them as well.
You’ll try to downgrade qubes-core-dom0 and it could require another 10 packages to downgrade and these 10 packages could require another 10 packages each to downgrade.
Maybe there is a way to list all the dependencies that you need to downgrade as well to downgrade specific package, but I don’t know of it.

Maybe you can get the list of installed packages since the last working ones from dnf history and just downgrade them all.
But this list could be quite big.