AX211 Fails to Start on a range of kernels/firmwares (5.18.9, latest)

User Support General
1

Hi There!
Trying to get (soldered) Intel ax211 card running on an X1 nano, gen2 and nothing I’ve tried seems to be working. Note that a Ubuntu 22.04 live-usb loads and allows the card to connect flawlessly under 5.15, so-a0-gf-a0-64.ucode, and pnvm 881c99e1.

I’ve tried the following configurations in a mixture of qvm-

  1. debian-11 template, kernels 5.15.x and 5.18.9, linux-firmware from 20220708 and 20220610.
  2. fedora-34 template, same kernel/firmware trials as above
  3. fedora-36 template, same kernel/firmware trials as above

I’ve also blacklisted iwl* modules in dom0. I’m probably too unfamiliar with pci devices but I’ve also tried the following:

  1. toggling no-strict-reset and permissive modes in qvm-pci
  2. hiding the device from xen-pciback and allowing xen-pciback passthrough.
  3. Disabling 11ax in iwlwifi

@marmarek I’m curious if you have any other suggestions for xen-pciback and/or can shed light if you think the differences in PCI capabilities between sys-net and dom0 are a reason for failure. Note that I’ve run this configuration before against the previous gen x1 nano with an ax210 and that was working fine. I’m also guessing that this Intel CNVI thing and moving some of the mac layer to the processor may have something to do with it.

Logs here since I’m a new user (long time listener, first time caller):

Thanks,
S

2

You can try debian 11 or fedora 36 live-usb and check if ax211 will work there and if it works then you can try to use this working kernel installed in debian-11/fedora-36 template for sys-net and check if ax211 will work with it:
Managing qube kernels | Qubes OS

3

Thanks for this, booted to a fedora-36 live usb as suggestion. The card comes up and associates to AP fine under 5.17.5 and ucode 68. When replicating this setup under Qubes/sys-net as an HVM, still fails.

I’m beginning to suspect this has something to do with cnvio.

4

Did you manage to get ax211 working? I’m considering buying the ThinkPad X1 nano 2nd gen but I’m concerned that the WiFi won’t work.

5

This will help you guys:

This will make it work, and hopefully with some new code updates from all the relevant camps (Intel, Linux, Xen), it will work out-of-the-box once again.

1 Like
6

Thanks, I wish it would. I think we’re talking about slightly different cards here, note my firmware is so-a0 instead of ty:

[    7.365332] iwlwifi 0000:00:11.0: Detected Intel(R) Wi-Fi 6E AX211 160MHz, REV=0x370
[    7.365447] thermal thermal_zone0: failed to read out thermal zone (-61)
[    7.771538] iwlwifi 0000:00:11.0: Timeout waiting for PNVM load!
[    7.771644] iwlwifi 0000:00:11.0: Failed to start RT ucode: -110
[    7.771734] iwlwifi 0000:00:11.0: WRT: Collecting data: ini trigger 13 fired (delay=0ms).
[    7.773688] iwlwifi 0000:00:11.0: Start IWL Error Log Dump:
[    7.773764] iwlwifi 0000:00:11.0: Transport status: 0x00000042, valid: 6
[    7.773849] iwlwifi 0000:00:11.0: Loaded firmware version: 67.8f59b80b.0 so-a0-gf-a0-67.ucode
[    7.773962] iwlwifi 0000:00:11.0: 0x00000084 | NMI_INTERRUPT_UNKNOWN       
[    7.774047] iwlwifi 0000:00:11.0: 0x000002F0 | trm_hw_status0
[    7.774126] iwlwifi 0000:00:11.0: 0x00000000 | trm_hw_status1
[    7.774203] iwlwifi 0000:00:11.0: 0x004D9EF2 | branchlink2
[    7.774270] iwlwifi 0000:00:11.0: 0x004D01BA | interruptlink1
[    7.774348] iwlwifi 0000:00:11.0: 0x004D01BA | interruptlink2

Which isn’t supported until 5.14. See here. I’ve tried the solution suggested above with 5.10 and 5.14, while removing the pnvm and/or putting different versions of the pnvm in there. No luck on anything, even recompiled the kernel to 8x the PNVM load timeout.

I’ve also tried up to 5.18.16 with bleeding edge linux-firmware. I’ve also tried a ton of xen pci-passthrough stuff from the GPU passthrough tutorials but haven’t had any luck there either. Looking at xl dmesg doesn’t point to any obvious PCIe access violations.

Not sure if anyone with PCI, ACPI, VT-d, and/or UEFI experience like @marmarek can weigh in here, but I’m stuck. After reading the iwlwifi kernel code, it also looks like Fedora and Ubuntu live CDs look to load PNVM from a UEFI variable which isn’t available in sys-net. So I’m wondering if the platform specific pnvm is the only thing that’ll load and boot the ax211 on this box. Interestingly, bluetooth loads just fine on same chipset.

Otherwise, if you turn up debug on modprobe iwlwifi debug=0xffffffff then you get a bunch of spam from iwl_pcie_restock_bd just before failure:

[  524.835864] iwlwifi 0000:00:11.0: iwl_pcie_restock_bd Assigned virtual RB ID 13 to queue 0 index 4088
[  524.835866] iwlwifi 0000:00:11.0: iwl_pcie_restock_bd Assigned virtual RB ID 12 to queue 0 index 4089
[  524.835868] iwlwifi 0000:00:11.0: iwl_pcie_restock_bd Assigned virtual RB ID 11 to queue 0 index 4090
[  524.835870] iwlwifi 0000:00:11.0: iwl_pcie_restock_bd Assigned virtual RB ID 10 to queue 0 index 4091
[  524.835872] iwlwifi 0000:00:11.0: iwl_pcie_restock_bd Assigned virtual RB ID 9 to queue 0 index 4092
[  524.835874] iwlwifi 0000:00:11.0: iwl_pcie_restock_bd Assigned virtual RB ID 8 to queue 0 index 4093
[  524.835876] iwlwifi 0000:00:11.0: iwl_pcie_restock_bd Assigned virtual RB ID 7 to queue 0 index 4094
[  524.835886] iwlwifi 0000:00:11.0: iwl_enable_fw_load_int_ctx_info Enabling ALIVE interrupt only
[  524.891906] iwlwifi 0000:00:11.0: iwl_pcie_irq_handler ISR inta 0x80000000, enabled 0x80000001(sw), enabled(hw) 0x00000000, fh 0x00010000
[  524.891957] iwlwifi 0000:00:11.0: iwl_pcie_irq_handler inta 0x80000000, enabled 0x00000000
[  524.891958] iwlwifi 0000:00:11.0: iwl_pcie_irq_handler Rx interrupt
[  524.891962] iwlwifi 0000:00:11.0: iwl_pcie_rx_handle Q 0: HW = 2, SW = 0
[  524.891964] iwlwifi 0000:00:11.0: iwl_pcie_get_rxb Got virtual RB ID 4101
[  524.891966] iwlwifi 0000:00:11.0: iwl_pcie_rx_handle_rb Q 0: cmd at offset 0: UCODE_ALIVE_NTFY (00. 1, seq 0xc000)
[  524.891971] iwlwifi 0000:00:11.0: iwl_alive_fn Got sku_id: 0x0610d1 0x00 0x00
[  524.891973] iwlwifi 0000:00:11.0: iwl_alive_fn Alive ucode status 0xcafe revision 0x1 0x0
[  524.891974] iwlwifi 0000:00:11.0: iwl_alive_fn Alive ucode CDB
[  524.891975] iwlwifi 0000:00:11.0: iwl_alive_fn UMAC version: Major - 0x19, Minor - 0x0
[  524.891977] iwlwifi 0000:00:11.0: iwl_pcie_rx_handle Q 0: HW = 2, SW = 1
[  524.891978] iwlwifi 0000:00:11.0: iwl_pcie_get_rxb Got virtual RB ID 4100
[  524.891979] iwlwifi 0000:00:11.0: iwl_pcie_rx_handle_rb Q 0: cmd at offset 0: MFUART_LOAD_NOTIFICATION (00.b1, seq 0xc001)
[  524.891981] iwlwifi 0000:00:11.0: iwl_mvm_rx_mfuart_notif MFUART: installed ver: 0x1400000c, external ver: 0x1400000c, status: 0x00010080, duration: 0x00000020
[  524.891983] iwlwifi 0000:00:11.0: iwl_mvm_rx_mfuart_notif MFUART: image size: 0x00006094
[  524.891984] iwlwifi 0000:00:11.0: iwl_pcie_restock_bd Assigned virtual RB ID 4101 to queue 0 index 4095
[  524.891985] iwlwifi 0000:00:11.0: iwl_pcie_restock_bd Assigned virtual RB ID 4100 to queue 0 index 0
[  524.891986] iwlwifi 0000:00:11.0: iwl_pcie_napi_poll [0] handled 2, budget 64
[  525.146817] iwlwifi 0000:00:11.0: Timeout waiting for PNVM load!
[  525.146857] iwlwifi 0000:00:11.0: Failed to start RT ucode: -110
[  525.146892] iwlwifi 0000:00:11.0: WRT: Collecting data: ini trigger 13 fired (delay=0ms).
[  525.146932] iwlwifi 0000:00:11.0: iwl_dbg_tlv_tp_trigger WRT: tp 13, reset_fw 2
[  525.146934] iwlwifi 0000:00:11.0: iwl_dbg_tlv_tp_trigger WRT: restart_required 0, last_tp_resetfw 255
[  525.146937] iwlwifi 0000:00:11.0: iwl_dbg_tlv_tp_trigger WRT: stop and reload firmware
[  525.148788] iwlwifi 0000:00:11.0: Start IWL Error Log Dump:
[  525.148811] iwlwifi 0000:00:11.0: Transport status: 0x00000042, valid: 6
[  525.148836] iwlwifi 0000:00:11.0: Loaded firmware version: 67.8f59b80b.0 so-a0-gf-a0-67.ucode
[  525.148868] iwlwifi 0000:00:11.0: 0x00000084 | NMI_INTERRUPT_UNKNOWN

@Azz19 I’ve put a ton of hours to get this working on the X1 nano gen2. I had a gen1 running qubes super well, but only had 16gb of ram. This would be an ideal qubes box once wifi is working. Right now I’m usb-wifiing it…

7

I wish your suggested solution would work. I think we’re talking about slightly different cards here, note my firmware is so-a0 instead of ty:

[    7.365332] iwlwifi 0000:00:11.0: Detected Intel(R) Wi-Fi 6E AX211 160MHz, REV=0x370
[    7.365447] thermal thermal_zone0: failed to read out thermal zone (-61)
[    7.771538] iwlwifi 0000:00:11.0: Timeout waiting for PNVM load!
[    7.771644] iwlwifi 0000:00:11.0: Failed to start RT ucode: -110
[    7.771734] iwlwifi 0000:00:11.0: WRT: Collecting data: ini trigger 13 fired (delay=0ms).
[    7.773688] iwlwifi 0000:00:11.0: Start IWL Error Log Dump:
[    7.773764] iwlwifi 0000:00:11.0: Transport status: 0x00000042, valid: 6
[    7.773849] iwlwifi 0000:00:11.0: Loaded firmware version: 67.8f59b80b.0 so-a0-gf-a0-67.ucode
[    7.773962] iwlwifi 0000:00:11.0: 0x00000084 | NMI_INTERRUPT_UNKNOWN       
[    7.774047] iwlwifi 0000:00:11.0: 0x000002F0 | trm_hw_status0
[    7.774126] iwlwifi 0000:00:11.0: 0x00000000 | trm_hw_status1
[    7.774203] iwlwifi 0000:00:11.0: 0x004D9EF2 | branchlink2
[    7.774270] iwlwifi 0000:00:11.0: 0x004D01BA | interruptlink1
[    7.774348] iwlwifi 0000:00:11.0: 0x004D01BA | interruptlink2

Which isn’t supported until 5.14. See here. I’ve tried the solution suggested above with 5.10 and 5.14, while removing the pnvm and/or putting different versions of the pnvm in there. No luck on anything, even recompiled the kernel to 8x the PNVM load timeout.

I’ve also tried up to 5.18.16 with bleeding edge linux-firmware. I’ve also tried a ton of xen pci-passthrough stuff from the GPU passthrough tutorials but haven’t had any luck there either. Looking at xl dmesg doesn’t point to any obvious PCIe access violations.

Not sure if anyone with PCI, ACPI, VT-d, and/or UEFI experience like @marmarek can weigh in here, but I’m stuck. After reading the iwlwifi kernel code, it also looks like Fedora and Ubuntu live CDs look to load PNVM from a UEFI variable which isn’t available in sys-net. So I’m wondering if the platform specific pnvm is the only thing that’ll load and boot the ax211 on this box. Interestingly, bluetooth loads just fine on same chipset.

Otherwise, if you turn up debug on modprobe iwlwifi debug=0xffffffff then you get a bunch of spam from iwl_pcie_restock_bd just before failure:

[  524.835864] iwlwifi 0000:00:11.0: iwl_pcie_restock_bd Assigned virtual RB ID 13 to queue 0 index 4088
[  524.835866] iwlwifi 0000:00:11.0: iwl_pcie_restock_bd Assigned virtual RB ID 12 to queue 0 index 4089
[  524.835868] iwlwifi 0000:00:11.0: iwl_pcie_restock_bd Assigned virtual RB ID 11 to queue 0 index 4090
[  524.835870] iwlwifi 0000:00:11.0: iwl_pcie_restock_bd Assigned virtual RB ID 10 to queue 0 index 4091
[  524.835872] iwlwifi 0000:00:11.0: iwl_pcie_restock_bd Assigned virtual RB ID 9 to queue 0 index 4092
[  524.835874] iwlwifi 0000:00:11.0: iwl_pcie_restock_bd Assigned virtual RB ID 8 to queue 0 index 4093
[  524.835876] iwlwifi 0000:00:11.0: iwl_pcie_restock_bd Assigned virtual RB ID 7 to queue 0 index 4094
[  524.835886] iwlwifi 0000:00:11.0: iwl_enable_fw_load_int_ctx_info Enabling ALIVE interrupt only
[  524.891906] iwlwifi 0000:00:11.0: iwl_pcie_irq_handler ISR inta 0x80000000, enabled 0x80000001(sw), enabled(hw) 0x00000000, fh 0x00010000
[  524.891957] iwlwifi 0000:00:11.0: iwl_pcie_irq_handler inta 0x80000000, enabled 0x00000000
[  524.891958] iwlwifi 0000:00:11.0: iwl_pcie_irq_handler Rx interrupt
[  524.891962] iwlwifi 0000:00:11.0: iwl_pcie_rx_handle Q 0: HW = 2, SW = 0
[  524.891964] iwlwifi 0000:00:11.0: iwl_pcie_get_rxb Got virtual RB ID 4101
[  524.891966] iwlwifi 0000:00:11.0: iwl_pcie_rx_handle_rb Q 0: cmd at offset 0: UCODE_ALIVE_NTFY (00. 1, seq 0xc000)
[  524.891971] iwlwifi 0000:00:11.0: iwl_alive_fn Got sku_id: 0x0610d1 0x00 0x00
[  524.891973] iwlwifi 0000:00:11.0: iwl_alive_fn Alive ucode status 0xcafe revision 0x1 0x0
[  524.891974] iwlwifi 0000:00:11.0: iwl_alive_fn Alive ucode CDB
[  524.891975] iwlwifi 0000:00:11.0: iwl_alive_fn UMAC version: Major - 0x19, Minor - 0x0
[  524.891977] iwlwifi 0000:00:11.0: iwl_pcie_rx_handle Q 0: HW = 2, SW = 1
[  524.891978] iwlwifi 0000:00:11.0: iwl_pcie_get_rxb Got virtual RB ID 4100
[  524.891979] iwlwifi 0000:00:11.0: iwl_pcie_rx_handle_rb Q 0: cmd at offset 0: MFUART_LOAD_NOTIFICATION (00.b1, seq 0xc001)
[  524.891981] iwlwifi 0000:00:11.0: iwl_mvm_rx_mfuart_notif MFUART: installed ver: 0x1400000c, external ver: 0x1400000c, status: 0x00010080, duration: 0x00000020
[  524.891983] iwlwifi 0000:00:11.0: iwl_mvm_rx_mfuart_notif MFUART: image size: 0x00006094
[  524.891984] iwlwifi 0000:00:11.0: iwl_pcie_restock_bd Assigned virtual RB ID 4101 to queue 0 index 4095
[  524.891985] iwlwifi 0000:00:11.0: iwl_pcie_restock_bd Assigned virtual RB ID 4100 to queue 0 index 0
[  524.891986] iwlwifi 0000:00:11.0: iwl_pcie_napi_poll [0] handled 2, budget 64
[  525.146817] iwlwifi 0000:00:11.0: Timeout waiting for PNVM load!
[  525.146857] iwlwifi 0000:00:11.0: Failed to start RT ucode: -110
[  525.146892] iwlwifi 0000:00:11.0: WRT: Collecting data: ini trigger 13 fired (delay=0ms).
[  525.146932] iwlwifi 0000:00:11.0: iwl_dbg_tlv_tp_trigger WRT: tp 13, reset_fw 2
[  525.146934] iwlwifi 0000:00:11.0: iwl_dbg_tlv_tp_trigger WRT: restart_required 0, last_tp_resetfw 255
[  525.146937] iwlwifi 0000:00:11.0: iwl_dbg_tlv_tp_trigger WRT: stop and reload firmware
[  525.148788] iwlwifi 0000:00:11.0: Start IWL Error Log Dump:
[  525.148811] iwlwifi 0000:00:11.0: Transport status: 0x00000042, valid: 6
[  525.148836] iwlwifi 0000:00:11.0: Loaded firmware version: 67.8f59b80b.0 so-a0-gf-a0-67.ucode
[  525.148868] iwlwifi 0000:00:11.0: 0x00000084 | NMI_INTERRUPT_UNKNOWN

@Azz19 I’ve put a ton of hours to get this working on the X1 nano gen2. I had a gen1 running qubes super well, but only had 16gb of ram. This would be an ideal qubes box once wifi is working. Right now I’m usb-wifiing it…

8

This may work on the AX211. On my AX210 I have only been able to get it to work by adding the “Settings” application to the Fedora sys-net Qube then

  1. Moving the relevant *.pnvm.xz file to *.bak in order to see WiFi networks
  2. Connect to the desired WiFi network (it will fail)
  3. Using sys-net “Settings” turn the Wireless OFF, then back ON (WiFi will then connect)

Odd, but it works.

9

Huh, interesting suggestion. Thanks. It doesn’t work for me, however, since the iwlwifi driver doesn’t even bring up the mac layer and boot the interface with/without the pnvm load. In working condition, booted outside of qubes, you’ll see 5.14+ spit out the mac addr of the card.

10

My bad….:see_no_evil:

This is why it’s failing.

But if the PNVM .xz file isn’t present, then it should work….

Is there a file called iwlwifi-so*.pnvm.xz?

If you move that file out of /lib/firmware and rmmod and modprobe iwlmvm and iwlwifi, what happens?

That’s basically the same as the steps I suggested. When you click those GUI buttons, they execute the same functions as those commands I said.

They’re just a more visually-pleasing way to do them :yum:

11

PNVM loading still timeout after removing all iwlwifi-so*.pnvm.xz with kernel 5.18.9

$ ls /lib/firmware/iwlwifi-so*pnvm*
/lib/firmware/iwlwifi-so-a0-gf-a0.pnvm.xz
/lib/firmware/iwlwifi-so-a0-gf4-a0.pnvm.xz

sudo rm /lib/firmware/iwlwifi-so*pnvm*
# Restart wifi
sudo rmmod iwlmvm
sudo rmmod iwlwifi
sudo modprobe iwlmvm

Log

$ dmesg | grep iwl
[    3.189787] iwlwifi 0000:00:06.0: Direct firmware load for iwlwifi-so-a0-gf-a0-72.ucode failed with error -2
[    3.217972] iwlwifi 0000:00:06.0: api flags index 2 larger than supported by driver
[    3.218000] iwlwifi 0000:00:06.0: TLV_FW_FSEQ_VERSION: FSEQ Version: 0.0.2.36
[    3.220075] iwlwifi 0000:00:06.0: loaded firmware version 71.058653f6.0 so-a0-gf-a0-71.ucode op_mode iwlmvm
[    3.516202] iwlwifi 0000:00:06.0: Detected Intel(R) Wi-Fi 6E AX211 160MHz, REV=0x430
[    3.692788] iwlwifi 0000:00:06.0: loaded PNVM version 881c99e1
[    3.951719] iwlwifi 0000:00:06.0: Timeout waiting for PNVM load!
[    3.951745] iwlwifi 0000:00:06.0: Failed to start RT ucode: -110
12

I’m entirely in the same boat as you. I’ve spent countless hours reading every post in every forum I can find on the ax210 (the card I have) in for any fixes to qubes errors (or linux in general, but every other distro seems to work every else now but Qubes). I came to the same judgment as you are right that its a virtualization issue with the PNVM, but removing it or keeping it doesn’t help, not do any older or newer kernels for me. I did have it working on kernel 5.10 a couple months ago, with a workaround but a recent update broke that too. Framework laptop here, so at least not soldered in, but still want it to work. I’m happy to help troubleshoot or test ideas if anyone has any that I haven’t tried already.

13

I wonder how you managed to get it working on kernel 5.10 if ax211 was only supported as of kernel 5.14 en:users:drivers:iwlwifi [Linux Wireless].

14

The truth is I don’t know. But to be clear, I am using the AX210, so there is some slight difference. I do know that I could get it on kernel 5.10 with the hacked workaround of unloading the iwlmvm and iwlwifi, and the reloading those on each boot. But ironically, I could never get them working on a later kernel, even though technically this card was supported only 5.11 onward! Very strange. It’s absolutely a Qubes issue, though, as I have tried this on 5 different linux distros in a whole range of kernels, and they all work easily (including Fedora which is the base of dom0). So something with the virtualization in Xen doesn’t pass through correctly these particular chipsets.

15

Indeed, a Qubes / XEN issue, see iwlwifi fails to load with model ax200-series · Issue #5615 · QubesOS/qubes-issues · GitHub

16

Ah, thanks for linking that! I am happy to try to test it when there is a live patch attempt. I know it did not work for me as recently as 2 weeks ago, so I assume the conversation you linked to did not result in a working patch attempt yet? Honestly not very familiar with how these patches are implemented.

17

I am also not familiar otherwise I would have tested, but at least I am happy to see that the issue is on the radar screen of those that are familiar.

18

@makuoly thanks a ton for pointing this patch out.

Thank you @crogers1 for making this available. As of this AM, I can confirm that the patch referenced here works verbatim:

For me, it works against a minor update to 5.18.19 (see version file) and insertion of the patch into series.conf and 'kernel.spec.in` from:

I’m sure the Github issue referenced could use some weigh in (2x link limit reached for me).

Happy to post RPMs for this kernel somewhere if that’s of interest to others that are frustrated with Intel on this thread.

19

Qubes ftamework user at official matrix room has reported that AX210 works with iwlwifi on 5.19 kernel. Can any framework user confirm?

20

I’d certainly like to hear about this as well. It’s important to note that the firmware issue is not linux kernel-related though (as far as I understand) so the kernel version should not change anything as long as it is after 5.11 when Intel officially supported it. The issue is with Xen itself, so that would need to be patched with that patch above? Whoever knows more, though, please let us know :slight_smile:

@scsich It sounds like you got the patch to work? Did you need to compile your own kernel? I think that’s what you were explaining above but I am not yet familiar enough with that process to be certain I understand.