[qubes-users] kernel-latest broke my system

In trying to make my wifi adapter working, I decided to try kernel-latest on Dom0, which installed kernel 5.10.11-1.fc25.qubes.x86_64. The result is a system where I cannot start VMs (not even VMs with no devices connected to them) due to libvirt errors ( The kernel doesn’t support reset from sysfs for PCI device …).

I tried to go back to my old kernels by changing xen.cfg in /boot/efi/EFI/qubes (here I have options 5.4.90-1.qubes.x86_64 and 5.4.91-1.fc25.qubes.x86_64, besides the one I mentioned above). The real big problem is that these kernels do not seem to appear to work anymore. As soon as I change default in xen.cgf selecting one of these two kernels, I am not able to access the system (after I insert the LUKS passphrase I get black screen in the authorization manager. Moreover, from boot messages it seems that neither these kernel can start sys-net anymore).

Any suggestion is really appreciated, I spent the last week configuring my PC and I would literally break into tears if I had to re-do everything from scratch.

Fabrizio Romano Genovese:

In trying to make my wifi adapter working, I decided to try `kernel-latest`
on Dom0, which installed kernel `5.10.11-1.fc25.qubes.x86_64`. The result
is a system where I cannot start VMs (not even VMs with no devices
connected to them) due to `libvirt` errors ( The kernel doesn't support
reset from sysfs for PCI device ...).

I tried to go back to my old kernels by changing `xen.cfg` in
`/boot/efi/EFI/qubes` (here I have options 5.4.90-1.qubes.x86_64 and
5.4.91-1.fc25.qubes.x86_64, besides the one I mentioned above). The real
big problem is that these kernels do not seem to appear to work anymore. As
soon as I change `default` in `xen.cgf` selecting one of these two kernels,
I am not able to access the system (after I insert the LUKS passphrase I
get black screen in the authorization manager. Moreover, from boot messages
it seems that neither these kernel can start sys-net anymore).

Any suggestion is really appreciated, I spent the last week configuring my
PC and I would literally break into tears if I had to re-do everything from
scratch.

Did you also install kernel-latest-qubes-vm (in addition to
kernel-latest) in dom0? Then maybe that too happens to be somehow
broken on your system.

If you can log in on a console (Ctrl-Alt-F2) *after* all your
autostart VMs have failed to start, check 'qubes-prefs default_kernel'
and try setting the VM kernel to another version - i.e. to one of the
directory names in /var/lib/qubes/vm-kernels/ - like this:

    qubes-prefs default_kernel 5.4.90-1
    qubes-prefs default_kernel 5.4.91-1.fc25

If you can't log in at all, you could mount the root filesystem from a
Qubes installer console and edit the 'default_kernel' property inside
var/lib/qubes/qubes.xml on the root filesystem mountpoint.

Rusty

Yes, I installed kernel-latest-qubes-vm as well, while still using 5.4.90-1 on dom0, and things ran smoothly without much problems, even after several restarts. I did what you suggested but nothing really happens.

It turns out this is a bug of kernel 5.10.11-1.fc25.qubes.x86_64, which is being discussed here: dom0 kernel 5.10.11-1.fc25 from current-testing -> libxenxl errors (revert to 5.10.8-1 resolves) · Issue #6377 · QubesOS/qubes-issues · GitHub
The bug will be probably fixed already in 5.10.12, the problem is that I don’t see how I can install the upgrade if I cannot launch VMs.

What really puzzles me is that 5.10.11-1 somehow managed to break also 5.4.90-1.qubes.x86_64 and 5.4.91-1.fc25.qubes.x86_64.

In any case, elaborating on your hints it turns out that using kernel 5.4.91-1 I can get a shell by doing Ctrl+Alt+F2. I am greeted by a message saying that X won’t start, but at least I can login.
Here I can launch VMs like vault, that have no device assigned (on kernel 5.10.11-1 I cannot launch anything).

Still no hope for sys-net: It says “Unable to reset PCI device : no FLR, PM reset or bus reset available, see /var/log/libvirt/libxl/libxl-driver.log for details”.
Following the suggestion, the log says "unable to add device with path ", and “libxl_devices_destroy failed for 5”.

…If I manage to get internet running, I may download the new, patched kernel and hopefully be back with a functioning computer. Any clue as what may be the next steps here?

…Following the suggestions on PCI Troubleshooting | Qubes OS I was able to start sys-net and get internet running. I’m running qubes-dom0-update and it works, but the new kernel version is not yet available. I hope that installing the new kernel solves the problem, it should be a matter of hours before it can be installed. Fingers crossed! If things go well (let’s hope!) I’ll document what I did in dom0 kernel 5.10.11-1.fc25 from current-testing -> libxenxl errors (revert to 5.10.8-1 resolves) · Issue #6377 · QubesOS/qubes-issues · GitHub in case other people end up in the same nasty situation. :slight_smile: