Yet another USB keyboard thread

adromanov · May 25, 2026, 1:21pm

@marmarek I have checked the latest 6.19.14-1.qubes.fc41.x86_64 - the crash is still there

[2026-05-25 21:08:31] [    0.000000] Linux version 6.19.14-1.qubes.fc41.x86_64 (mockbuild@1f1f754d3dcc40af9e866af6d0b35c71) (gcc (GCC) 14.3.1 20251022 (Red Hat 14.3.1-4), GNU ld version 2.43.1-14.fc41) #1 SMP PREEMPT_DYNAMIC Tue Apr 28 18:06:16 UTC 2026
...
[2026-05-25 21:08:31] [    0.419800] BUG: kernel NULL pointer dereference, address: 0000000000000000
[2026-05-25 21:08:31] [    0.419820] #PF: supervisor read access in kernel mode
[2026-05-25 21:08:31] [    0.419833] #PF: error_code(0x0000) - not-present page
[2026-05-25 21:08:31] [    0.419846] PGD 0 P4D 0 
[2026-05-25 21:08:31] [    0.419855] Oops: Oops: 0000 [#1] SMP NOPTI
[2026-05-25 21:08:31] [    0.419866] CPU: 0 UID: 0 PID: 1 Comm: swapper/0 Not tainted 6.19.14-1.qubes.fc41.x86_64 #1 PREEMPT(full) 
[2026-05-25 21:08:31] [    0.419887] Hardware name: Xen HVM domU, BIOS 4.19.4 05/12/2026
[2026-05-25 21:08:31] [    0.419902] RIP: 0010:__amd_smn_rw+0x30/0x100
[2026-05-25 21:08:31] [    0.419918] Code: 05 41 30 c9 01 66 0f af 05 ad 2f c9 01 41 57 41 56 41 55 41 54 55 53 66 39 c2 0f 83 c0 00 00 00 48 8b 05 03 fd 06 03 0f b7 d2 <4c> 8b 34 d0 4d 85 f6 0f 84 a9 00 00 00 80 3d e4 fc 06 03 00 0f 84
[2026-05-25 21:08:31] [    0.419956] RSP: 0018:ffffd4fe8001fd58 EFLAGS: 00010297
[2026-05-25 21:08:31] [    0.419969] RAX: 0000000000000000 RBX: ffffd4fe8001fda4 RCX: 0000000010136008
[2026-05-25 21:08:31] [    0.419987] RDX: 0000000000000000 RSI: 0000000000000064 RDI: 0000000000000060
[2026-05-25 21:08:31] [    0.420005] RBP: ffffffffb5b6ada0 R08: ffffd4fe8001fda4 R09: 0000000000000000
[2026-05-25 21:08:32] [    0.420023] R10: ffffffffb696aa40 R11: 0000000000000020 R12: ffff8e5b8243a000
[2026-05-25 21:08:32] [    0.420040] R13: ffffffffb66ab3b0 R14: 000000000cf33959 R15: ffff8e5b8243a000
[2026-05-25 21:08:32] [    0.420060] FS:  0000000000000000(0000) GS:ffff8e5bd92cb000(0000) knlGS:0000000000000000
[2026-05-25 21:08:32] [    0.420078] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[2026-05-25 21:08:32] [    0.420093] CR2: 0000000000000000 CR3: 0000000006c2e000 CR4: 0000000000750ef0
[2026-05-25 21:08:32] [    0.420113] PKRU: 55555554
[2026-05-25 21:08:32] [    0.420120] Call Trace:
[2026-05-25 21:08:32] [    0.420129]  <TASK>
[2026-05-25 21:08:32] [    0.420137]  ? __pfx_quirk_clear_strap_no_soft_reset_dev2_f0+0x10/0x10
[2026-05-25 21:08:32] [    0.420155]  amd_smn_read+0x27/0x50
[2026-05-25 21:08:32] [    0.420167]  quirk_clear_strap_no_soft_reset_dev2_f0+0x37/0x80
[2026-05-25 21:08:32] [    0.420183]  pci_fixup_device+0xf9/0x1b0
[2026-05-25 21:08:32] [    0.420196]  pci_apply_final_quirks+0xe9/0x280
[2026-05-25 21:08:32] [    0.420211]  ? __pfx_pci_apply_final_quirks+0x10/0x10
[2026-05-25 21:08:32] [    0.420225]  do_one_initcall+0x5a/0x310
[2026-05-25 21:08:32] [    0.420237]  do_initcalls+0x233/0x290
[2026-05-25 21:08:32] [    0.420250]  kernel_init_freeable+0x18a/0x210
[2026-05-25 21:08:32] [    0.420264]  ? __pfx_kernel_init+0x10/0x10
[2026-05-25 21:08:32] [    0.420276]  kernel_init+0x1a/0x140
[2026-05-25 21:08:32] [    0.420287]  ret_from_fork+0x126/0x190
[2026-05-25 21:08:32] [    0.420299]  ? __pfx_kernel_init+0x10/0x10
[2026-05-25 21:08:32] [    0.420310]  ret_from_fork_asm+0x1a/0x30
[2026-05-25 21:08:32] [    0.420323]  </TASK>
[2026-05-25 21:08:32] [    0.420330] Modules linked in:
[2026-05-25 21:08:32] [    0.420342] CR2: 0000000000000000
[2026-05-25 21:08:32] [    0.420352] ---[ end trace 0000000000000000 ]---
[2026-05-25 21:08:32] [    0.420365] RIP: 0010:__amd_smn_rw+0x30/0x100
[2026-05-25 21:08:32] [    0.420379] Code: 05 41 30 c9 01 66 0f af 05 ad 2f c9 01 41 57 41 56 41 55 41 54 55 53 66 39 c2 0f 83 c0 00 00 00 48 8b 05 03 fd 06 03 0f b7 d2 <4c> 8b 34 d0 4d 85 f6 0f 84 a9 00 00 00 80 3d e4 fc 06 03 00 0f 84
[2026-05-25 21:08:32] [    0.420417] RSP: 0018:ffffd4fe8001fd58 EFLAGS: 00010297
[2026-05-25 21:08:32] [    0.420430] RAX: 0000000000000000 RBX: ffffd4fe8001fda4 RCX: 0000000010136008
[2026-05-25 21:08:32] [    0.420448] RDX: 0000000000000000 RSI: 0000000000000064 RDI: 0000000000000060
[2026-05-25 21:08:32] [    0.420465] RBP: ffffffffb5b6ada0 R08: ffffd4fe8001fda4 R09: 0000000000000000
[2026-05-25 21:08:32] [    0.420483] R10: ffffffffb696aa40 R11: 0000000000000020 R12: ffff8e5b8243a000
[2026-05-25 21:08:32] [    0.420500] R13: ffffffffb66ab3b0 R14: 000000000cf33959 R15: ffff8e5b8243a000
[2026-05-25 21:08:32] [    0.420561] FS:  0000000000000000(0000) GS:ffff8e5bd92cb000(0000) knlGS:0000000000000000
[2026-05-25 21:08:32] [    0.420561] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[2026-05-25 21:08:32] [    0.420561] CR2: 0000000000000000 CR3: 0000000006c2e000 CR4: 0000000000750ef0
[2026-05-25 21:08:32] [    0.420561] PKRU: 55555554
[2026-05-25 21:08:32] [    0.420561] Kernel panic - not syncing: Fatal exception

Is patch included in that kernel version? If yes, does not seem to help. If no - could you please upload kernel with the patch applied and I will test.

My hardware is MSI Tomahawk X870E + 9950x3D. I have the same crash on 6.18 and no crash on 6.12

adromanov · May 30, 2026, 3:33pm

Not sure how to edit the post (or if that is even possible), so I’ll add a new reply.
@marmarek I manually applied the patch to the kernel 6.18.29 (to minimize the scope), no more kernel panic. sys-usb and sys-audio both seems to work fine on the patched kernel, at least keyboard\mouse\sound.

[2026-05-30 23:18:48] [    0.000000] Linux version 6.18.29-123.qubes.fc41.x86_64 (mockbuild@826ea3c283a8) (gcc (GCC) 14.3.1 20251022 (Red Hat 14.3.1-4), GNU ld version 2.43.1-14.fc41) #1 SMP PREEMPT_DYNAMIC Sat May 30 11:40:26 UTC 2026
...
[2026-05-30 23:18:49] [    0.418808] pci 0000:00:00.0: Limiting direct PCI/PCI transfers
[2026-05-30 23:18:49] [    0.423180] AMD SMN roots not initialized.
[2026-05-30 23:18:49] [    0.423193] pci 0000:00:0c.0: Failed to read data
[2026-05-30 23:18:49] [    0.423248] PCI: CLS 0 bytes, default 64

marmarek · June 4, 2026, 11:12am

@adromanov do be clear - 6.18.29 without the patch still crashed, right? It already has a67540127042c736faf614497a7480f197638464 included, but not the other patch.

marmarek · June 5, 2026, 8:29pm

There is an alternative set of patches at Making sure you're not a bot!

I’ve pushed them to Apply patches for sys-usb crash on some AMD by marmarek · Pull Request #1304 · QubesOS/qubes-linux-kernel · GitHub

Can somebody with affected hardware test this version, please?

adromanov · June 7, 2026, 8:07am

do be clear - 6.18.29 without the patch still crashed, right?

@marmarek Correct, 6.18.29 crashes as well as 6.19.14.

There is an alternative set of patches at … Can somebody with affected hardware test this version, please?

This set of patches from https://github.com/marmarek/qubes-linux-kernel/tree/test-usb-crash also works:

[2026-06-07 15:41:38] [    0.000000] Linux version 6.18.31-1.qubes.fc41.x86_64 (mockbuild@7c9203a22d4b) (gcc (GCC) 14.3.1 20251022 (Red Hat 14.3.1-4), GNU ld version 2.43.1-14.fc41) #1 SMP PREEMPT_DYNAMIC Sun Jun  7 07:17:39 UTC 2026
...
[2026-06-07 15:41:40] [    1.438220] systemd[1]: Detected virtualization xen.
[2026-06-07 15:41:40] [    1.438242] systemd[1]: Detected architecture x86-64.
[2026-06-07 15:41:40] 
[2026-06-07 15:41:40] Welcome to Debian GNU/Linux 13 (trixie)!

But I’m not sure why kernel version 6.18.31-1.qubes.fc41.x86_64 (mockbuild@7c9203a22d4b) shows commit 7c9203a22d4b instead the one from your repo. I would assume it should be 9d37c7c:

╭─user@qubes-dev ~/qubes-builderv2/artifacts/sources/linux-kernel (9d37c7c)
╰─$ git remote -v
marmarek	https://github.com/marmarek/qubes-linux-kernel (fetch)
marmarek	https://github.com/marmarek/qubes-linux-kernel (push)
origin	https://github.com/QubesOS/qubes-linux-kernel (fetch)
origin	https://github.com/QubesOS/qubes-linux-kernel (push)

╭─user@qubes-dev ~/qubes-builderv2/artifacts/sources/linux-kernel (9d37c7c)
╰─$ git status
HEAD detached at marmarek/test-usb-crash
nothing to commit, working tree clean

╭─user@qubes-dev ~/qubes-builderv2/artifacts/sources/linux-kernel (9d37c7c)
╰─$ git log -1
commit 9d37c7c8a55d57085efe8128a40aad2355d0dd28 (HEAD, marmarek/test-usb-crash)
Author: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
Date:   Fri Jun 5 22:27:55 2026 +0200

    Apply patches for sys-usb crash on some AMD
    
    https://lore.kernel.org/all/20260602184823.GKah8ld2QJLm28xoa9@fat_crate.local/T/#u
    
    https://forum.qubes-os.org/t/yet-another-usb-keyboard-thread/38355/23

╭─user@qubes-dev ~/qubes-builderv2/artifacts/sources/linux-kernel (9d37c7c)
╰─$ git show 7c9203a22d4b
fatal: ambiguous argument '7c9203a22d4b': unknown revision or path not in the working tree.

In builder.yml I have:

components:
  - linux-kernel:
      url: file:///home/user/qubes-builderv2/artifacts/sources/linux-kernel
      branch: HEAD

Is that expected?

marmarek · June 7, 2026, 11:38am

That’s hostname, which happens to be container id used by mock.

CrisF · July 22, 2026, 9:50am

With the help of AI I was able to overcome some issues setting up 4.3.1 on a Minisforum MS-A2 having issues mentioned in this thread. the AI suggested I post here in the hope it would help others. I hope it does.
—
Confirming the quirk_clear_strap_no_soft_reset_dev2_f0 → amd_smn_read NULL-pointer panic on additional hardware, with version test results, since testing was requested in this thread / on LKML.

Hardware: Minisforum MS-A2, Ryzen 9 9955HX (Granite Ridge / Fire Range, family 0x1a model 0x44), BIOS 1.03 (CPU:9000, 2026/04/01). Passed-through device: AMD Raphael/Granite Ridge USB 2.0 xHCI (also reproduced with the machine’s USB 3.1 xHCI assigned).

Software: Qubes OS 4.3.1, Xen 4.19.4, sys-usb as HVM with PCI passthrough, strict-reset disabled for the controller.

Results by guest (VM) kernel:

6.18.35-1.qubes.fc41 — panic at boot (trace below)
7.0.14-1.qubes.fc41 — identical panic, identical trace (below)
6.12.64-1.qubes.fc41 — boots and works correctly; sys-usb enumerates devices (Bluetooth, mass storage) and has been stable across reboots with autostart enabled

This matches Marek’s LKML report exactly (worked in 6.12.59, broken by 6.17.x): the SMN accessor faults because amd_roots is never initialized in a domU where the USB controller is passed through without the AMD root complex. Host is unaffected; only the guest panics.

I have not yet tested the proposed patch (adbf61cc47cb / the amd_node.c NULL guard) but am happy to test a build if useful. Currently pinned to 6.12.64 for sys-usb (via qvm-prefs sys-usb kernel) with everything else on 6.18.35, which is a fully working configuration.

Guest kernel 6.18.35-1.qubes.fc41 (from /var/log/xen/console/guest-sys-usb.log; register dump and Code: bytes omitted):

pci 0000:00:01.0: PIIX3: Enabling Passive Release
pci 0000:00:01.0: Limiting direct PCI/PCI transfers
BUG: kernel NULL pointer dereference, address: 0000000000000000
#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
PGD 0 P4D 0
Oops: Oops: 0000 [#1] SMP NOPTI
CPU: 0 UID: 0 PID: 1 Comm: swapper/0 Not tainted 6.18.35-1.qubes.fc41.x86_64 #1 PREEMPT(full)
Hardware name: Xen HVM domU, BIOS 4.19.4 06/09/2026
RIP: 0010:__amd_smn_rw+0x30/0x100
Call Trace:
 <TASK>
 ? __pfx_quirk_clear_strap_no_soft_reset_dev2_f0+0x10/0x10
 amd_smn_read+0x27/0x50
 quirk_clear_strap_no_soft_reset_dev2_f0+0x37/0x80
 pci_fixup_device+0xf9/0x1b0
 pci_apply_final_quirks+0xe9/0x280
 ? __pfx_pci_apply_final_quirks+0x10/0x10
 do_one_initcall+0x5a/0x310
 do_initcalls+0x1ef/0x240
 kernel_init_freeable+0x18a/0x210
 ? __pfx_kernel_init+0x10/0x10
 kernel_init+0x1a/0x140
 ret_from_fork+0x158/0x170
 ? __pfx_kernel_init+0x10/0x10
 ret_from_fork_asm+0x1a/0x30
 </TASK>
Modules linked in:
CR2: 0000000000000000
Kernel panic - not syncing: Fatal exception

Guest kernel 7.0.14-1.qubes.fc41 — identical failure:

BUG: kernel NULL pointer dereference, address: 0000000000000000
#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
PGD 0 P4D 0
Oops: Oops: 0000 [#1] SMP NOPTI
CPU: 1 UID: 0 PID: 1 Comm: swapper/0 Not tainted 7.0.14-1.qubes.fc41.x86_64 #1 PREEMPT(full)
Hardware name: Xen HVM domU, BIOS 4.19.4 06/09/2026
RIP: 0010:__amd_smn_rw+0x30/0x100
Call Trace:
 <TASK>
 ? __pfx_quirk_clear_strap_no_soft_reset_dev2_f0+0x10/0x10
 amd_smn_read+0x27/0x50
 quirk_clear_strap_no_soft_reset_dev2_f0+0x37/0x80
 pci_fixup_device+0xf9/0x1b0
 pci_apply_final_quirks+0xe9/0x280
 ? __pfx_pci_apply_final_quirks+0x10/0x10
 do_one_initcall+0x5c/0x330
 do_initcalls+0x233/0x290
 kernel_init_freeable+0x186/0x230
 ? __pfx_kernel_init+0x10/0x10
 kernel_init+0x1a/0x140
 ret_from_fork+0x19e/0x260
 ? __pfx_kernel_init+0x10/0x10
 ret_from_fork_asm+0x1a/0x30
 </TASK>
Modules linked in:
CR2: 0000000000000000
Kernel panic - not syncing: Fatal exception

Full photos of both traces (including register dumps) available on request.

indietonic · July 22, 2026, 10:09am

I was able to build your branch using qubes-builderv2.

Now, how do I go about installing the kernel for sys-usb? Do I need to copy over the rpm files to dom0? Forgive me if this is documented somewhere, I was unable to find it.

Edit: I got it to work. I can confirm the patches work to resolve the issue.