Today I woke up to my qubes 4.2.4 machine giving me a green screen. The entire monitor was showing just the color green. I tried to break out of it, re-connected the HDMI cable, power-cycled the monitor, etc. So finally I gave up and force-rebooted my machine.
When I booted back up, the screen was working fine, but my keyboard wasn’t working, so i figured it must be a problem with my sys-usb qube. I rebooted with qubes.skip_autostart in the kernel options and this fixed my keyboard.
Now debugging the sys-usb qube, I checked its logs in /var/log/xen/console/guest-sys-usb.log, and found this:
Logs
[2026-04-11 09:10:12] [ 0.213879] pci 0000:00:01.1: BAR 4 [io 0x1000-0x100f]: assigned
[2026-04-11 09:10:12] [ 0.213879] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]
[2026-04-11 09:10:12] [ 0.213879] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]
[2026-04-11 09:10:12] [ 0.213879] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]
[2026-04-11 09:10:12] [ 0.213879] pci_bus 0000:00: resource 7 [mem 0xf0000000-0xfbffffff window]
[2026-04-11 09:10:12] [ 0.213879] pci 0000:00:01.0: PIIX3: Enabling Passive Release
[2026-04-11 09:10:12] [ 0.213879] pci 0000:00:00.0: Limiting direct PCI/PCI transfers
[2026-04-11 09:10:12] [ 0.221634] pci 0000:00:06.0: quirk_usb_early_handoff+0x0/0x1c0 took 13897 usecs
[2026-04-11 09:10:12] [ 0.224735] BUG: kernel NULL pointer dereference, address: 0000000000000000
[2026-04-11 09:10:12] [ 0.224751] #PF: supervisor read access in kernel mode
[2026-04-11 09:10:12] [ 0.224764] #PF: error_code(0x0000) - not-present page
[2026-04-11 09:10:12] [ 0.224778] PGD 0 P4D 0
[2026-04-11 09:10:12] [ 0.224791] Oops: Oops: 0000 [#1] SMP NOPTI
[2026-04-11 09:10:12] [ 0.224803] CPU: 0 UID: 0 PID: 1 Comm: swapper/0 Not tainted 6.18.15-1.qubes.fc37.x86_64 #1 PREEMPT(full)
[2026-04-11 09:10:12] [ 0.224825] Hardware name: Xen HVM domU, BIOS 4.17.6 03/17/2026
[2026-04-11 09:10:12] [ 0.224840] RIP: 0010:__amd_smn_rw+0x30/0x100
[2026-04-11 09:10:12] [ 0.224856] Code: 05 ad 2a c3 01 66 0f af 05 1d 2a c3 01 41 57 41 56 41 55 41 54 55 53 66 39 c2 0f 83 d3 00 00 00 48 8b 05 73 67 03 03 0f b7 d2 <4c> 8b 3c d0 4d 85 ff 0f 84 bc 00 00 00 80 3d 54 67 03 03 00 0f 84
[2026-04-11 09:10:12] [ 0.225281] RSP: 0018:ffffd1090001fd48 EFLAGS: 00010297
[2026-04-11 09:10:12] [ 0.225281] RAX: 0000000000000000 RBX: ffffd1090001fd94 RCX: 0000000010136008
[2026-04-11 09:10:12] [ 0.225281] RDX: 0000000000000000 RSI: 0000000000000064 RDI: 0000000000000060
[2026-04-11 09:10:12] [ 0.225281] RBP: ffffffffa3050c90 R08: ffffd1090001fd94 R09: 0000000000000000
[2026-04-11 09:10:12] [ 0.225281] R10: ffffffffa3f68360 R11: 0000000000000020 R12: ffff8b93c1ede000
[2026-04-11 09:10:12] [ 0.225281] R13: ffffffffa3baeb40 R14: 00000000059239ff R15: ffff8b93c1ede000
[2026-04-11 09:10:12] [ 0.225281] FS: 0000000000000000(0000) GS:ffff8b9432175000(0000) knlGS:0000000000000000
[2026-04-11 09:10:12] [ 0.225281] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[2026-04-11 09:10:12] [ 0.225281] CR2: 0000000000000000 CR3: 000000001362c000 CR4: 0000000000750ef0
[2026-04-11 09:10:12] [ 0.225281] PKRU: 55555554
[2026-04-11 09:10:12] [ 0.225281] Call Trace:
[2026-04-11 09:10:12] [ 0.225281] <TASK>
[2026-04-11 09:10:12] [ 0.225281] ? __pfx_quirk_clear_strap_no_soft_reset_dev2_f0+0x10/0x10
[2026-04-11 09:10:12] [ 0.225281] amd_smn_read+0x27/0x50
[2026-04-11 09:10:12] [ 0.225281] quirk_clear_strap_no_soft_reset_dev2_f0+0x39/0xb0
[2026-04-11 09:10:12] [ 0.225281] pci_do_fixups+0x96/0x120
[2026-04-11 09:10:12] [ 0.225281] pci_apply_final_quirks+0xe8/0x2c0
[2026-04-11 09:10:12] [ 0.225281] ? __pfx_pci_apply_final_quirks+0x10/0x10
[2026-04-11 09:10:12] [ 0.225281] do_one_initcall+0x5a/0x330
[2026-04-11 09:10:12] [ 0.225281] do_initcalls+0x194/0x330
[2026-04-11 09:10:12] [ 0.225281] kernel_init_freeable+0x1e7/0x370
[2026-04-11 09:10:12] [ 0.225281] ? __pfx_kernel_init+0x10/0x10
[2026-04-11 09:10:12] [ 0.225281] kernel_init+0x1a/0x1d0
[2026-04-11 09:10:12] [ 0.225281] ret_from_fork+0xf2/0x110
[2026-04-11 09:10:12] [ 0.225281] ? __pfx_kernel_init+0x10/0x10
[2026-04-11 09:10:12] [ 0.225281] ret_from_fork_asm+0x1a/0x30
[2026-04-11 09:10:12] [ 0.225281] </TASK>
[2026-04-11 09:10:12] [ 0.225281] Modules linked in:
[2026-04-11 09:10:12] [ 0.225281] CR2: 0000000000000000
[2026-04-11 09:10:12] [ 0.225281] ---[ end trace 0000000000000000 ]---
[2026-04-11 09:10:12] [ 0.225281] RIP: 0010:__amd_smn_rw+0x30/0x100
[2026-04-11 09:10:12] [ 0.225281] Code: 05 ad 2a c3 01 66 0f af 05 1d 2a c3 01 41 57 41 56 41 55 41 54 55 53 66 39 c2 0f 83 d3 00 00 00 48 8b 05 73 67 03 03 0f b7 d2 <4c> 8b 3c d0 4d 85 ff 0f 84 bc 00 00 00 80 3d 54 67 03 03 00 0f 84
[2026-04-11 09:10:12] [ 0.225281] RSP: 0018:ffffd1090001fd48 EFLAGS: 00010297
[2026-04-11 09:10:12] [ 0.225281] RAX: 0000000000000000 RBX: ffffd1090001fd94 RCX: 0000000010136008
[2026-04-11 09:10:12] [ 0.225281] RDX: 0000000000000000 RSI: 0000000000000064 RDI: 0000000000000060
[2026-04-11 09:10:12] [ 0.225281] RBP: ffffffffa3050c90 R08: ffffd1090001fd94 R09: 0000000000000000
[2026-04-11 09:10:12] [ 0.225281] R10: ffffffffa3f68360 R11: 0000000000000020 R12: ffff8b93c1ede000
[2026-04-11 09:10:12] [ 0.225281] R13: ffffffffa3baeb40 R14: 00000000059239ff R15: ffff8b93c1ede000
[2026-04-11 09:10:12] [ 0.225281] FS: 0000000000000000(0000) GS:ffff8b9432175000(0000) knlGS:0000000000000000
[2026-04-11 09:10:12] [ 0.225281] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[2026-04-11 09:10:12] [ 0.225281] CR2: 0000000000000000 CR3: 000000001362c000 CR4: 0000000000750ef0
[2026-04-11 09:10:12] [ 0.225281] PKRU: 55555554
[2026-04-11 09:10:12] [ 0.225281] Kernel panic - not syncing: Fatal exception
The key line was this:
[2026-04-11 09:10:12] [ 0.224735] BUG: kernel NULL pointer dereference, address: 0000000000000000
After some trial and error, I found the panic only occurs if a specific USB controller on my machine is attached via PCI passthrough to sys-usb. My machine has 4 USB controllers overall, so I reconnected my mouse and keyboard to one of the three still-functional controllers to continue debugging.
I created a new qube named test-usb, set it up just like sys-usb (HVM mode, no mem-balancing), and then i attached this borked USB controller to test-usb. This qube also crashes with the exact same kernel panic.
After more trial & error, I found the panic only occurs when I have linux kernel version 6.18.15-1.fc37 loaded in the test-usb qube. If I switch to 6.12.63-1.fc37 or 6.12.64-1.fc37, then test-usb can load without issue. So in theory, I should be able to fix it by switching sys-usb to use linux kernel version 6.12. However i’d prefer to be able to run newer kernel versions if possible.
For the record, my dom0 is using kernel version 6.19.5-1.qubes.fc37.x86_64.
Could this be a kernel bug or something wrong with my system setup?