Unable to enter LUKS password using USB keyboard after kernel upgrade

After updating my dom0 kernel to kernel-latest (v6.12) , when I boot my computer the USB keyboard is completely unresponsive. The result is that I cannot enter the LUKS password to the decrypt the hard drive and continue the boot process. The keyboard works with GRUB. Since the computer does not have a PS/2 keyboard port (USB only), the only way to get out of that is to forcibly reset the system.

If I boot with an older kernel (6.11), the USB keyboard works and I can boot, so that’s what I have been doing. But I would like to know how to fix this before the older kernels get pushed out of my system.

Any ideas will be appreciated, thanks!

Do you see any usb-related errors when booting 6.12? If you can’t switch to detailed view with ESC key, you can remove rhgb and quiet options from kernel params in grub.

Is there something special about your hardware? Is that an extra USB controller on a PCIe card, or the standard one built into CPU/chipset?

I am using an extra USB controller on a PCIe card (where only a known keyboard and mouse are plugged in), so I can keep that one in dom0, while all the other USB ports go to sys-usb.

I booted without the ‘rhgb’ and ‘quiet’ parameters in the kernel command line, but the lines go too fast, and the framebuffer still gets initialized so I lose access to the earlier messages (nothing about USB in the ones I can see, they are mostly related to the GPU). I tried taking a video of the boot, but it’s too blurry to read. I’ll keep trying to see if I can get those messages, but if you have any other ideas to try, I’d be grateful.

Thanks!

You can make messages flow slower by adding boot_delay=5 to kernel options.

I finally managed to get a capture of the boot log.

When booting with 6.11. I see two USB controllers initialized, while with 6.12 only one is initialized, the other one (which seems to be the PCI card) seems missing. I wonder if something may have changed in the kernel command line and the USB card is not being passed to dom0? But there are no errors I can see, it seems to be just silently failing.

The other question I now have is why there is an extra USB controller in dom0, which should have been left for sys-usb. I’ll have to go and check my kernel command line later (I’m not at that computer right now).

I can upload the captured images, although I don’t know if that will provide a lot of information.

The other question I now have is why there is an extra USB controller in dom0

The answer is “Duh, because I told the kernel to do so.” :slight_smile:
So that is not a problem anymore.

I think I found the issue. It seems that in 6.12 there was a change to the Renesas PCI USB driver (what my PCI USB card seems to be), see Making sure you're not a bot!
There is some discussion at Gentoo Forums :: View topic - [SOLVED] 6.12.8 issues with USB3 (XHCI_PCI_RENESAS)

The kernel configs for 6.12 and 6.13 have

CONFIG_USB_XHCI_PCI_RENESAS=m
CONFIG_USB_XHCI_PLATFORM=m

so the driver must be compiled, but somehow it does not seem to load. Could it be a ramdisk issue with the modules?

I have a Renesas usb card on my desk, about to be installed…
Looking at modprobe on the xhci-pci-renesas modules for 6.11 and later kernels, I can see there is a mention of firmware file for 6.12 and later: renesas_usb_fw.mem

Do you see any Dom0 kernel messages about missing firmware? Normally it would be in /lib/firmware, I think, but there is nothing I can see here. Maybe the newer kernels can only work with this firmware file.

I see a message about a missing firwmare with kernel 6.11, something to the effect of “firmware not found, using ROM”. It would find it surprising that if the board has a ROM firmware it won’t work with an external firmware file. But I do not see any errors about missing firmware with kernel 6.12 and up, it seems to just not be there.

It is strange.
Can you see the module in Dom0, using lsmod ? Maybe you are correct, and the module has not been included in the initrd file. Maybe there is logic in dracut to verify the firmware is found, and to avoid modules with missing firmware, even though the device has ROM.
If it is not there, have you tried loading the module using modprobe <path to module> ? I am not expert enough to know if this should work in Qubes, or if it can cause any problems, but I would try it on standard linux. If it does load the driver, then some method is needed to force the module to be included in the initrd.
(Note: before, I did modinfo to see the firmware information in the ko file, not modprobe)

I forgot to say - and you probably know - it is also possible to mount the initrd file, and look at what modules are included.

I can’t see if the module is loaded, nor load it manually because the system is stuck in the LUKS password stage. I need to find out if it’s possible to boot into a shell in the ramdisk to explore a bit more. Inspecting the initrd file would also be a good idea…

xhci-pci-renesas is probably not included in the initrd. You can add it by creating a /etc/dracut.cond.d/user.conf file with one line:

add_drivers+=" xhci-pci-renesas "

and then regenerate initrd with dracut -f --kver FULL_KERNEL_VERSION /boot/initramfs-FULL_KERNEL_VERSION.img (replace FULL_KERNEL_VERSION with appropriate version, you can take it from the file name.

I went through a lot of pain with USB just recently. Your problem sounds a little bit similar.
Do you have rd.qubes.hide_all_usb set on your /vmlinuz line in grub ? I am guessing you also have a rd.qubes.dom0_usb= line, which allowed your Renesas device to be visible in Dom0. If you edit the grub line to delete the hide_all_usb then maybe you can log in. Then you could follow Marek’s advice, I hope.

Here are my notes about the effect of different kernel commandline grub items:

  • rd.qubes.hide_all_usb
    There will be no USB devices at boot. It will be impossible to type the encryption password if only USB kb.
  • rd.qubes.hide_pci=00.0.0
    Hide the device from dom0.
  • rd.qubes.dom0_usb=00.0.0
    Given a PCI device, then expose it to dom0.
    Overrides rd.qubes.hide_all_usb for this one device.
    Not clear that it must be a usb pci dev - could also enable a network dev ?
    Enables usbguard, so only kb/mouse are active in Dom0.

[edit: the ‘00.0.0’ must be changed to match a specific PCI device.

That was exactly the problem. Regenerating the initrd with that extra driver made it work. Thank you very much!

It seems that the renesas USB driver was not included anymore by default in the initrd after the changes (if I understood correctly, it used to be a dependency of the generic xhci-pci driver, and that’s why it would be automatically included).

Any chances to get this added to the official OS images? This would simplify installation for people using these kinds of cards. I got lucky I could install with the older kernel.

1 Like

Yes, lets track it here: Unable to enter LUKS password using USB keyboard after kernel upgrade · Issue #9889 · QubesOS/qubes-issues · GitHub

1 Like