Libxenlight failed to create new domain "sys-usb"

TL;DR

libxenlight cannot create sys-usb when rebooting it after (what seems like) any update. If it fails that way, I cannot start sys-usb manually.

It starts correctly on boot or if I shutdown sys-usb myself.

This state is fixed by rebooting or suspending the computer.

Log

Here is what my /var/log/libvirt/libxl/libxl-driver.log says about that:

2024-09-25 05:01:54.156+0000: libxl: libxl_pci.c:1587:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:00:0d.0
2024-09-25 05:01:54.166+0000: libxl: libxl_pci.c:1587:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:00:0d.2
2024-09-25 05:01:55.258+0000: libxl: libxl_qmp.c:1837:qmp_ev_parse_error_messages: Domain 81:Could not open '/sys/bus/pci/devices/0000:00:00.0/config': No such file or directory
2024-09-25 05:01:55.258+0000: libxl: libxl_pci.c:1275:pci_add_qmp_device_add_cb: Domain 81:Retrying PCI add 1
2024-09-25 05:01:56.321+0000: libxl: libxl_qmp.c:1837:qmp_ev_parse_error_messages: Domain 81:Could not open '/sys/bus/pci/devices/0000:00:00.0/config': No such file or directory
2024-09-25 05:01:56.322+0000: libxl: libxl_pci.c:1275:pci_add_qmp_device_add_cb: Domain 81:Retrying PCI add 2
2024-09-25 05:01:57.386+0000: libxl: libxl_qmp.c:1837:qmp_ev_parse_error_messages: Domain 81:Could not open '/sys/bus/pci/devices/0000:00:00.0/config': No such file or directory
2024-09-25 05:01:57.386+0000: libxl: libxl_pci.c:1275:pci_add_qmp_device_add_cb: Domain 81:Retrying PCI add 3
2024-09-25 05:01:58.450+0000: libxl: libxl_qmp.c:1837:qmp_ev_parse_error_messages: Domain 81:Could not open '/sys/bus/pci/devices/0000:00:00.0/config': No such file or directory
2024-09-25 05:01:58.450+0000: libxl: libxl_pci.c:1275:pci_add_qmp_device_add_cb: Domain 81:Retrying PCI add 4
2024-09-25 05:01:59.518+0000: libxl: libxl_qmp.c:1837:qmp_ev_parse_error_messages: Domain 81:Could not open '/sys/bus/pci/devices/0000:00:00.0/config': No such file or directory
2024-09-25 05:01:59.518+0000: libxl: libxl_pci.c:1275:pci_add_qmp_device_add_cb: Domain 81:Retrying PCI add 5
2024-09-25 05:02:00.607+0000: libxl: libxl_qmp.c:1837:qmp_ev_parse_error_messages: Domain 81:Could not open '/sys/bus/pci/devices/0000:00:00.0/config': No such file or directory
2024-09-25 05:02:00.609+0000: libxl: libxl_pci.c:1275:pci_add_qmp_device_add_cb: Domain 81:Retrying PCI add 6
2024-09-25 05:02:01.674+0000: libxl: libxl_qmp.c:1837:qmp_ev_parse_error_messages: Domain 81:Could not open '/sys/bus/pci/devices/0000:00:00.0/config': No such file or directory
2024-09-25 05:02:01.674+0000: libxl: libxl_pci.c:1275:pci_add_qmp_device_add_cb: Domain 81:Retrying PCI add 7
2024-09-25 05:02:02.737+0000: libxl: libxl_qmp.c:1837:qmp_ev_parse_error_messages: Domain 81:Could not open '/sys/bus/pci/devices/0000:00:00.0/config': No such file or directory
2024-09-25 05:02:02.738+0000: libxl: libxl_pci.c:1275:pci_add_qmp_device_add_cb: Domain 81:Retrying PCI add 8
2024-09-25 05:02:03.802+0000: libxl: libxl_qmp.c:1837:qmp_ev_parse_error_messages: Domain 81:Could not open '/sys/bus/pci/devices/0000:00:00.0/config': No such file or directory
2024-09-25 05:02:03.802+0000: libxl: libxl_pci.c:1275:pci_add_qmp_device_add_cb: Domain 81:Retrying PCI add 9
2024-09-25 05:02:04.866+0000: libxl: libxl_qmp.c:1837:qmp_ev_parse_error_messages: Domain 81:Could not open '/sys/bus/pci/devices/0000:00:00.0/config': No such file or directory
2024-09-25 05:02:04.866+0000: libxl: libxl_pci.c:1275:pci_add_qmp_device_add_cb: Domain 81:Retrying PCI add 10
2024-09-25 05:02:05.168+0000: libxl: libxl_pci.c:1864:device_pci_add_done: Domain 81:libxl__device_pci_add failed for PCI device 0:0:d.0 (rc -9)
2024-09-25 05:02:05.169+0000: libxl: libxl_pci.c:1864:device_pci_add_done: Domain 81:libxl__device_pci_add failed for PCI device 0:0:d.2 (rc -9)
2024-09-25 05:02:05.169+0000: libxl: libxl_create.c:2000:domcreate_attach_devices: Domain 81:unable to add pci devices
2024-09-25 05:02:05.171+0000: libxl: libxl_pci.c:1587:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:00:0d.0
2024-09-25 05:02:15.204+0000: libxl: libxl_device.c:1489:libxl__wait_for_backend: Backend /local/domain/0/backend/pci/82/0 not ready
2024-09-25 05:02:15.205+0000: libxl: libxl_pci.c:1587:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:00:0d.2
2024-09-25 05:02:25.237+0000: libxl: libxl_device.c:1489:libxl__wait_for_backend: Backend /local/domain/0/backend/pci/82/0 not ready
2024-09-25 05:02:25.378+0000: libxl: libxl_exec.c:120:libxl_report_child_exitstatus: killed fork (dying as expected) [358552] unexpectedly exited status zero
2024-09-25 05:04:06.621+0000: libxl: libxl_pci.c:1587:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:01:00.1
2024-09-25 05:05:20.558+0000: libxl: libxl_pci.c:1587:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:00:0d.0
2024-09-25 05:05:20.573+0000: libxl: libxl_pci.c:1587:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:00:0d.2
2024-09-25 05:05:20.577+0000: libxl: libxl_pci.c:1587:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:00:14.0
2024-09-25 05:06:23.025+0000: libxl: libxl_pci.c:1587:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:01:00.1

I do not know why it complains about resets btw. I have no strict reset enabled.

Any suggestions?

Check the logs in var/log/xen/console/guest-sys-usb.log and var/log/xen/console/guest-sys-usb-dm.log, maybe there will be some info.

Maybe related issues:

guest-sys-usb-dm.log

There is a lot in here, looking into startup attempt at 05:31…

By now, I have found this:

[2024-09-25 05:00:32] xen:swiotlb_xen: Warning: only able to allocate 4 MB for software IO TLB
[2024-09-25 05:00:32] xen:swiotlb_xen: Lowering to 2MB
[2024-09-25 05:00:32] xen:swiotlb_xen: Lowering to 2MB
[2024-09-25 05:00:32] xen:swiotlb_xen: Lowering to 2MB
[2024-09-25 05:00:32] xen:swiotlb_xen: Failed to get contiguous memory for DMA from Xen!
[2024-09-25 05:00:32] You either: don't have the permissions, do not have enough free memory under 4GB, or the hypervisor memory is too fragmented! (rc:-12)
[2024-09-25 05:00:32] pcifront pci-0: Could not setup SWIOTLB!
[2024-09-25 05:00:32] pcifront pci-0: 12 Error setting up PCI Frontend
[2024-09-25 05:00:32] blkfront: xvda: flush diskcache: enabled; persistent grants: enabled; indirect descriptors: enabled; bounce buffer: enabled
[2024-09-25 05:00:32]  xvda: xvda1 xvda2 xvda3
[2024-09-25 05:00:32] pcifront pci-0: backend going away!

guest-sys-usb.log

guest-sys-usb.log has pretty much nothing in it:

[2024-09-25 04:59:01] Logfile Opened
[2024-09-25 05:00:31] Logfile Opened
[2024-09-25 05:01:53] Logfile Opened
[2024-09-25 05:05:20] Logfile Opened
[2024-09-25 05:05:24] [    0.000000] Linux version 6.6.48-1.qubes.fc37.x86_64 (mockbuild@065a31b3c1ba4c34ab3938416488814f) (gcc (GCC) 12.3.1 20230508 (Red Hat 12.3.1-1), GNU ld version 2.38-27.fc37) #1 SMP PREEMPT_DYNAMIC Wed Sep  4 01:09:59 GMT 2024

I think it have started correctly at 05:05:20, so everything below should be irrelevant.

Github issues

unman’s issue seems to be somewhat similar, and yet they can start the qube manually. Maybe it’s due to differences in hardware.

icequbes1 has logs in their issue, but I cannot find my version of Assigning real physical device 04:00.0 at all. It just fails to get contiguous memory for DMA from Xen!


I am going to read guest-sys-usb-dm.log more later, and maybe add more information to unman’s issue.

Try to set swiotlb=8192 kernel option for sys-usb: