My sys-usb, after its template updated to Debian 12.11 fails to start. Template is debian-12, updated through qubes-update-gui, requesting restart from there (which I usually do not do to avoid risking losing my devices).
2025-05-25 17:05:14.950+0000: libxl: libxl_qmp.c:1837:qmp_ev_parse_error_messages: Domain 24:Could not open '/sys/bus/pci/devices/0000:00:00.3/config': No such file or directory
2025-05-25 17:05:14.950+0000: libxl: libxl_pci.c:1275:pci_add_qmp_device_add_cb: Domain 24:Retrying PCI add 10
2025-05-25 17:05:15.276+0000: libxl: libxl_pci.c:1864:device_pci_add_done: Domain 24:libxl__device_pci_add failed for PCI device 0:7:0.3 (rc -9)
2025-05-25 17:05:15.277+0000: libxl: libxl_pci.c:1864:device_pci_add_done: Domain 24:libxl__device_pci_add failed for PCI device 0:7:0.4 (rc -9)
2025-05-25 17:05:15.277+0000: libxl: libxl_create.c:2000:domcreate_attach_devices: Domain 24:unable to add pci devices
I was initially confused about this 0000:00:00.3 which never existed on the host… I guess is the PCI id in the stubdom, to which the host’s 0000:07:00.3 device is passed-through?
A similar post makes me look at swiotlb, and sure enough when the guest fails the guest subdom kernel complains:
[2025-05-25 18:51:49] xen:swiotlb_xen: Warning: only able to allocate 4 MB for software IO TLB
[2025-05-25 18:51:49] xen:swiotlb_xen: Lowering to 2MB
[2025-05-25 18:51:49] xen:swiotlb_xen: Lowering to 2MB
[2025-05-25 18:51:49] xen:swiotlb_xen: Lowering to 2MB
[2025-05-25 18:51:49] xen:swiotlb_xen: Failed to get contiguous memory for DMA from Xen!
[2025-05-25 18:51:49] You either: don't have the permissions, do not have enough free memory under 4GB, or the hypervisor memory is too fragmented! (rc:-12)
[2025-05-25 18:51:49] pcifront pci-0: Could not setup SWIOTLB!
[2025-05-25 18:51:49] pcifront pci-0: 12 Error setting up PCI Frontend
Sure enough this did not happen on previous VM boot:
[2025-05-13 10:36:23] xen:swiotlb_xen: Warning: only able to allocate 4 MB for software IO TLB
[2025-05-13 10:36:23] software IO TLB: mapped [mem 0x0000000003000000-0x0000000003400000] (4MB)
Am I correct that this pcifront failure to initialize is the reason why passthrough configuration will necessarily fail? I guess this could be tested early to produce a more understandable error to the user. Opened issue for that aspect.
I tried to set swiotlb=8192 as suggested in that other thread. Unlike that other post, I still have the same errors referring to a non-existent PCI device, and the Could not setup SWIOTLB! error is still there.
Note the stubdom logs also are a bit more verbose:
[2025-05-25 19:05:14] from-vchan: {"execute":"device_add","id":2020372746,"arguments":{"driver":"xen-pci-passthrough","id":"pci-pt-07_00.3","hostaddr":"0000:07:00.3"}}
[2025-05-25 19:05:14]
[2025-05-25 19:05:14] from-unix: {"id": 2020372746, "error": {"class": "GenericError", "desc": "Could not open '/sys/bus/pci/devices/0000:00:00.3/config': No such file or directory"}}
[2025-05-25 19:05:14]
So… maybe I do not have enough free memory under 4GB, or the hypervisor memory is too fragmented - do we have an easy way to check that?