Libxenlight failed to create new domain "sys-usb"

otter2 · September 25, 2024, 6:47am

TL;DR

libxenlight cannot create sys-usb when rebooting it after (what seems like) any update. If it fails that way, I cannot start sys-usb manually.

It starts correctly on boot or if I shutdown sys-usb myself.

This state is fixed by rebooting or suspending the computer.

Log

Here is what my /var/log/libvirt/libxl/libxl-driver.log says about that:

2024-09-25 05:01:54.156+0000: libxl: libxl_pci.c:1587:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:00:0d.0
2024-09-25 05:01:54.166+0000: libxl: libxl_pci.c:1587:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:00:0d.2
2024-09-25 05:01:55.258+0000: libxl: libxl_qmp.c:1837:qmp_ev_parse_error_messages: Domain 81:Could not open '/sys/bus/pci/devices/0000:00:00.0/config': No such file or directory
2024-09-25 05:01:55.258+0000: libxl: libxl_pci.c:1275:pci_add_qmp_device_add_cb: Domain 81:Retrying PCI add 1
2024-09-25 05:01:56.321+0000: libxl: libxl_qmp.c:1837:qmp_ev_parse_error_messages: Domain 81:Could not open '/sys/bus/pci/devices/0000:00:00.0/config': No such file or directory
2024-09-25 05:01:56.322+0000: libxl: libxl_pci.c:1275:pci_add_qmp_device_add_cb: Domain 81:Retrying PCI add 2
2024-09-25 05:01:57.386+0000: libxl: libxl_qmp.c:1837:qmp_ev_parse_error_messages: Domain 81:Could not open '/sys/bus/pci/devices/0000:00:00.0/config': No such file or directory
2024-09-25 05:01:57.386+0000: libxl: libxl_pci.c:1275:pci_add_qmp_device_add_cb: Domain 81:Retrying PCI add 3
2024-09-25 05:01:58.450+0000: libxl: libxl_qmp.c:1837:qmp_ev_parse_error_messages: Domain 81:Could not open '/sys/bus/pci/devices/0000:00:00.0/config': No such file or directory
2024-09-25 05:01:58.450+0000: libxl: libxl_pci.c:1275:pci_add_qmp_device_add_cb: Domain 81:Retrying PCI add 4
2024-09-25 05:01:59.518+0000: libxl: libxl_qmp.c:1837:qmp_ev_parse_error_messages: Domain 81:Could not open '/sys/bus/pci/devices/0000:00:00.0/config': No such file or directory
2024-09-25 05:01:59.518+0000: libxl: libxl_pci.c:1275:pci_add_qmp_device_add_cb: Domain 81:Retrying PCI add 5
2024-09-25 05:02:00.607+0000: libxl: libxl_qmp.c:1837:qmp_ev_parse_error_messages: Domain 81:Could not open '/sys/bus/pci/devices/0000:00:00.0/config': No such file or directory
2024-09-25 05:02:00.609+0000: libxl: libxl_pci.c:1275:pci_add_qmp_device_add_cb: Domain 81:Retrying PCI add 6
2024-09-25 05:02:01.674+0000: libxl: libxl_qmp.c:1837:qmp_ev_parse_error_messages: Domain 81:Could not open '/sys/bus/pci/devices/0000:00:00.0/config': No such file or directory
2024-09-25 05:02:01.674+0000: libxl: libxl_pci.c:1275:pci_add_qmp_device_add_cb: Domain 81:Retrying PCI add 7
2024-09-25 05:02:02.737+0000: libxl: libxl_qmp.c:1837:qmp_ev_parse_error_messages: Domain 81:Could not open '/sys/bus/pci/devices/0000:00:00.0/config': No such file or directory
2024-09-25 05:02:02.738+0000: libxl: libxl_pci.c:1275:pci_add_qmp_device_add_cb: Domain 81:Retrying PCI add 8
2024-09-25 05:02:03.802+0000: libxl: libxl_qmp.c:1837:qmp_ev_parse_error_messages: Domain 81:Could not open '/sys/bus/pci/devices/0000:00:00.0/config': No such file or directory
2024-09-25 05:02:03.802+0000: libxl: libxl_pci.c:1275:pci_add_qmp_device_add_cb: Domain 81:Retrying PCI add 9
2024-09-25 05:02:04.866+0000: libxl: libxl_qmp.c:1837:qmp_ev_parse_error_messages: Domain 81:Could not open '/sys/bus/pci/devices/0000:00:00.0/config': No such file or directory
2024-09-25 05:02:04.866+0000: libxl: libxl_pci.c:1275:pci_add_qmp_device_add_cb: Domain 81:Retrying PCI add 10
2024-09-25 05:02:05.168+0000: libxl: libxl_pci.c:1864:device_pci_add_done: Domain 81:libxl__device_pci_add failed for PCI device 0:0:d.0 (rc -9)
2024-09-25 05:02:05.169+0000: libxl: libxl_pci.c:1864:device_pci_add_done: Domain 81:libxl__device_pci_add failed for PCI device 0:0:d.2 (rc -9)
2024-09-25 05:02:05.169+0000: libxl: libxl_create.c:2000:domcreate_attach_devices: Domain 81:unable to add pci devices
2024-09-25 05:02:05.171+0000: libxl: libxl_pci.c:1587:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:00:0d.0
2024-09-25 05:02:15.204+0000: libxl: libxl_device.c:1489:libxl__wait_for_backend: Backend /local/domain/0/backend/pci/82/0 not ready
2024-09-25 05:02:15.205+0000: libxl: libxl_pci.c:1587:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:00:0d.2
2024-09-25 05:02:25.237+0000: libxl: libxl_device.c:1489:libxl__wait_for_backend: Backend /local/domain/0/backend/pci/82/0 not ready
2024-09-25 05:02:25.378+0000: libxl: libxl_exec.c:120:libxl_report_child_exitstatus: killed fork (dying as expected) [358552] unexpectedly exited status zero
2024-09-25 05:04:06.621+0000: libxl: libxl_pci.c:1587:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:01:00.1
2024-09-25 05:05:20.558+0000: libxl: libxl_pci.c:1587:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:00:0d.0
2024-09-25 05:05:20.573+0000: libxl: libxl_pci.c:1587:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:00:0d.2
2024-09-25 05:05:20.577+0000: libxl: libxl_pci.c:1587:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:00:14.0
2024-09-25 05:06:23.025+0000: libxl: libxl_pci.c:1587:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:01:00.1

I do not know why it complains about resets btw. I have no strict reset enabled.

Any suggestions?

apparatus · September 25, 2024, 7:05am

Check the logs in var/log/xen/console/guest-sys-usb.log and var/log/xen/console/guest-sys-usb-dm.log, maybe there will be some info.

Maybe related issues:

github.com/QubesOS/qubes-issues

libxl boot error for HVM with PCI device attached, says /sys/.../0000:00:00.0/config doesn't exist

opened 02:14AM - 16 Nov 20 UTC

closed 05:05AM - 08 Jan 22 UTC

icequbes1

T: bug C: Xen P: default r4.1-buster-stable r4.1-bullseye-stable r4.1-dom0-stable diagnosed r4.1-centos-stream8-stable r4.1-bookworm-stable

**Qubes OS version** R4.1; xen-4.14.0-7.fc32; dom0 and Fedora 32 templates usin…g `current-testing` repos. **Affected component(s) or functionality** vmm-xen? core-libvirt? vmm-xen-stubdom-linux? libxl? **Brief summary** sys-net HVM qube fails to boot at times, resulting in a libxl error indicating `/sys/bus/pci/devices/0000:00:00.0/config` doesn't exist (within the stubdom): ``` 2020-11-15 19:06:33.880+0000: libxl: libxl_qmp.c:1836:qmp_ev_parse_error_messages: Domain 1:Could not open '/sys/bus/pci/devices/0000:00:00.0/config': No such file or directory 2020-11-15 19:06:33.883+0000: libxl: libxl_pci.c:1735:device_pci_add_done: Domain 1:libxl__device_pci_add failed for PCI device 0:4:0.0 (rc -28) 2020-11-15 19:06:33.883+0000: libxl: libxl_create.c:1898:domcreate_attach_devices: Domain 1:unable to add pci devices ``` **How Reproducible** Sometimes, perhaps 1 out of every 5 boots of sys-net. Not consistently. sys-net qube works fine when it does boot. **To Reproduce** Steps to reproduce the behavior: 1. Boot sys-net (or an HVM with PCI devices attached) **Expected behavior** Qube boots successfully via a "[qube] has started" UI toast. **Actual behavior** Qube fails to boot, instructing the user to refer to /var/log/libvirt/libxl/libxl-driver.log **Additional context** First occurrence of this error according to libxl-driver.log was October 31 2020. According to dnf.rpm.log, Xen was upgraded to 4.14.0-6 October 20 2020, but my logs only go back to mid September 2020 when the system was first installed. Based strictly on logs and without knowledge of libxl/libvirt/xen internals, perhaps there is some sort of race condition between installation of the PCI frontend in the stubdom and attempts to assign the PCI device to that controller? Failed log below shows "Assigning real physical device 04:00.0" and two entries later is "pcifront pci-0: Creating PCI Frontend Bus 0000:00" ``` [2020-11-15 11:06:33] pcifront pci-0: Installing PCI frontend^M [2020-11-15 11:06:33] xen:swiotlb_xen: Warning: only able to allocate 4 MB for software IO TLB^M [2020-11-15 11:06:33] software IO TLB: mapped [mem 0x05800000-0x05c00000] (4MB)^M [2020-11-15 11:06:33] written 110 bytes to vchan [2020-11-15 11:06:33] written 34 bytes to vchan [2020-11-15 11:06:33] [00:06.0] xen_pt_realize: Assigning real physical device 04:00.0 to devfn 0x30 [2020-11-15 11:06:33] written 152 bytes to vchan [2020-11-15 11:06:34] pcifront pci-0: Creating PCI Frontend Bus 0000:00^M [2020-11-15 11:06:34] pcifront pci-0: PCI host bridge to bus 0000:00^M [2020-11-15 11:06:34] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]^M [2020-11-15 11:06:34] pci_bus 0000:00: root bus resource [mem 0x00000000-0x7fffffffff]^M [2020-11-15 11:06:34] pci_bus 0000:00: root bus resource [bus 00-ff]^M [2020-11-15 11:06:34] pci 0000:00:00.0: [8086:24f3] type 00 class 0x028000^M [2020-11-15 11:06:34] pci 0000:00:00.0: reg 0x10: [mem 0xf1000000-0xf1001fff 64bit]^M [2020-11-15 11:06:34] pcifront pci-0: claiming resource 0000:00:00.0/0^M [2020-11-15 11:06:34] pcifront pci-0: Rescanning PCI Frontend Bus 0000:00^M ``` sys-net-dm log of a successful boot, I see "Assigning real physical device 04:00.0" only after all the PCI 0000:00 bus entries: ``` [2020-11-11 11:38:19] pcifront pci-0: Installing PCI frontend [2020-11-11 11:38:19] xen:swiotlb_xen: Warning: only able to allocate 4 MB for software IO TLB [2020-11-11 11:38:19] software IO TLB: mapped [mem 0x05800000-0x05c00000] (4MB) [2020-11-11 11:38:19] written 110 bytes to vchan [2020-11-11 11:38:19] pcifront pci-0: Creating PCI Frontend Bus 0000:00 [2020-11-11 11:38:19] pcifront pci-0: PCI host bridge to bus 0000:00 [2020-11-11 11:38:19] pci_bus 0000:00: root bus resource [io 0x0000-0xffff] [2020-11-11 11:38:19] pci_bus 0000:00: root bus resource [mem 0x00000000-0x7fffffffff] [2020-11-11 11:38:19] pci_bus 0000:00: root bus resource [bus 00-ff] [2020-11-11 11:38:19] pci 0000:00:00.0: [8086:24f3] type 00 class 0x028000 [2020-11-11 11:38:19] pci 0000:00:00.0: reg 0x10: [mem 0xf1000000-0xf1001fff 64bit] [2020-11-11 11:38:19] pcifront pci-0: claiming resource 0000:00:00.0/0 [2020-11-11 11:38:19] written 34 bytes to vchan [2020-11-11 11:38:19] [00:06.0] xen_pt_realize: Assigning real physical device 04:00.0 to devfn 0x30 [2020-11-11 11:38:19] [00:06.0] xen_pt_register_regions: IO region 0 registered (size=0x00002000 base_addr=0xf1000000 type: 0x4) [2020-11-11 11:38:19] [00:06.0] xen_pt_config_reg_init: Offset 0x000e mismatch! Emulated=0x0080, host=0x0000, syncing to 0x0000. [2020-11-11 11:38:19] [00:06.0] xen_pt_config_reg_init: Offset 0x0010 mismatch! Emulated=0x0000, host=0xf1000004, syncing to 0xf1000004. [2020-11-11 11:38:20] [00:06.0] xen_pt_config_reg_init: Offset 0x00ca mismatch! Emulated=0x0000, host=0x0023, syncing to 0x0023. [2020-11-11 11:38:20] [00:06.0] xen_pt_config_reg_init: Offset 0x00d2 mismatch! Emulated=0x0000, host=0x0080, syncing to 0x0080. [2020-11-11 11:38:20] [00:06.0] xen_pt_config_reg_init: Offset 0x0044 mismatch! Emulated=0x0000, host=0x10008ec0, syncing to 0x8ec0. [2020-11-11 11:38:20] [00:06.0] xen_pt_config_reg_init: Offset 0x004a mismatch! Emulated=0x0000, host=0x0010, syncing to 0x0010. [2020-11-11 11:38:20] [00:06.0] xen_pt_config_reg_init: Offset 0x0052 mismatch! Emulated=0x0000, host=0x1011, syncing to 0x1011. [2020-11-11 11:38:20] [00:06.0] xen_pt_pci_intx: intx=1 [2020-11-11 11:38:20] [00:06.0] xen_pt_realize: Real physical device 04:00.0 registered successfully [2020-11-11 11:38:20] written 34 bytes to vchan [2020-11-11 11:38:20] written 2048 bytes to vchan [2020-11-11 11:38:20] written 845 bytes to vchan [2020-11-11 11:38:20] written 110 bytes to vchan [2020-11-11 11:38:20] written 34 bytes to vchan [2020-11-11 11:38:20] written 34 bytes to vchan ``` HVMs in R4.1 do boot significantly slower as compared to R4.0, perhaps there is a race condition somewhere, but I wouldn't know if that is Qubes. Xen, libvirt, or a qemu issue. **Solutions you've tried** Simply attempting to boot the qube again usually resolves the issue. *Related, [non-duplicate](https://www.qubes-os.org/doc/reporting-bugs/#new-issues-should-not-be-duplicates-of-existing-issues) issues** Various 'fail to boot' issues exist, but I haven't seen any where the libxl-driver.log shows `/sys/.../0000:00:00.0/config` as the culprit.

github.com/QubesOS/qubes-issues

Sys-net failed to restart after template update

opened 08:19AM - 27 Dec 23 UTC

deeplow

T: bug C: manager/widget P: default needs diagnosis C: updates affects-4.2

[How to file a helpful issue](https://www.qubes-os.org/doc/issue-tracking/) #…## Qubes OS release 4.2 ### Brief summary Sys-net failed to restart on new updater. ### Steps to reproduce 1. I started the computer at around 7h50 AM (things running normally, internet working) 2. I ran the updater (updated non-admin qubes) and select "restart qubes" (sys-net, sys-whonix, sys-usb) 3. At around 7h13m sys-net failed to start with error (details bellow) 4. I manually started sys-net and it worked as usual ### Expected behavior Updater restarts the sys-net successfully ### Actual behavior Updater failed to restart sys-net, leaving the system without internet connection ### Logs and Details #### `journalctl` ``` Dec 27 08:06:56 dom0 kernel: audit: type=1130 audit(1703664416.659:447): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-tmpfiles-clean comm="systemd" exe="/usr/lib/systemd/systemd"> Dec 27 08:06:56 dom0 kernel: audit: type=1131 audit(1703664416.659:448): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-tmpfiles-clean comm="systemd" exe="/usr/lib/systemd/systemd"> Dec 27 08:07:11 dom0 lvm[1332]: No longer monitoring thin pool qubes_dom0-vm--pool-tpool. Dec 27 08:07:12 dom0 lvm[1332]: Monitoring thin pool qubes_dom0-vm--pool-tpool. Dec 27 08:07:13 dom0 lvm[1332]: No longer monitoring thin pool qubes_dom0-vm--pool-tpool. Dec 27 08:07:13 dom0 lvm[1332]: Monitoring thin pool qubes_dom0-vm--pool-tpool. Dec 27 08:07:14 dom0 libvirtd[1840]: internal error: Unable to reset PCI device 0000:00:0d.0: no FLR, PM reset or bus reset available Dec 27 08:07:14 dom0 libvirtd[1840]: internal error: Unable to reset PCI device 0000:00:0d.2: internal error: Unable to reset PCI device 0000:00:0d.0: no FLR, PM reset or bus reset available Dec 27 08:07:14 dom0 libvirtd[1840]: internal error: Unable to reset PCI device 0000:00:14.0: internal error: Unable to reset PCI device 0000:00:0d.2: internal error: Unable to reset PCI device > Dec 27 08:07:14 dom0 lvm[1332]: No longer monitoring thin pool qubes_dom0-vm--pool-tpool. Dec 27 08:07:15 dom0 lvm[1332]: Monitoring thin pool qubes_dom0-vm--pool-tpool. Dec 27 08:07:16 dom0 lvm[1332]: No longer monitoring thin pool qubes_dom0-vm--pool-tpool. Dec 27 08:07:17 dom0 lvm[1332]: Monitoring thin pool qubes_dom0-vm--pool-tpool. Dec 27 08:07:18 dom0 qubesd[3435]: vm.sys-net: Starting sys-net Dec 27 08:07:18 dom0 qubesd[3435]: socket.send() raised exception. Dec 27 08:07:18 dom0 qubesd[3435]: socket.send() raised exception. Dec 27 08:07:18 dom0 lvm[1332]: No longer monitoring thin pool qubes_dom0-vm--pool-tpool. Dec 27 08:07:18 dom0 lvm[1332]: Monitoring thin pool qubes_dom0-vm--pool-tpool. Dec 27 08:07:18 dom0 lvm[1332]: No longer monitoring thin pool qubes_dom0-vm--pool-tpool. Dec 27 08:07:18 dom0 lvm[1332]: Monitoring thin pool qubes_dom0-vm--pool-tpool. Dec 27 08:07:19 dom0 kernel: loop0: detected capacity change from 0 to 1045872 Dec 27 08:07:19 dom0 kernel: loop1: detected capacity change from 0 to 1045872 Dec 27 08:07:19 dom0 kernel: pciback 0000:a5:00.0: xen_pciback: vpci: assign to virtual slot 0 Dec 27 08:07:19 dom0 kernel: pciback 0000:a5:00.0: registering for 16 Dec 27 08:07:19 dom0 kernel: xen-blkback: backend/vbd/16/51712: using 1 queues, protocol 1 (x86_64-abi) persistent grants Dec 27 08:07:19 dom0 kernel: xen-blkback: backend/vbd/16/51728: using 1 queues, protocol 1 (x86_64-abi) persistent grants Dec 27 08:07:19 dom0 kernel: xen-blkback: backend/vbd/16/51744: using 1 queues, protocol 1 (x86_64-abi) persistent grants Dec 27 08:07:19 dom0 kernel: xen-blkback: backend/vbd/16/51760: using 1 queues, protocol 1 (x86_64-abi) persistent grants Dec 27 08:07:40 dom0 libvirtd[1840]: internal error: libxenlight failed to create new domain 'sys-net' Dec 27 08:07:40 dom0 qubesd[3435]: vm.sys-net: Start failed: internal error: libxenlight failed to create new domain 'sys-net' Dec 27 08:07:40 dom0 lvm[1332]: No longer monitoring thin pool qubes_dom0-vm--pool-tpool. Dec 27 08:07:41 dom0 lvm[1332]: Monitoring thin pool qubes_dom0-vm--pool-tpool. Dec 27 08:07:41 dom0 lvm[1332]: No longer monitoring thin pool qubes_dom0-vm--pool-tpool. Dec 27 08:07:41 dom0 lvm[1332]: Monitoring thin pool qubes_dom0-vm--pool-tpool. Dec 27 08:07:42 dom0 widget-wrapper[6228]: Exception in thread Thread-4 (perform_restart): Dec 27 08:07:42 dom0 widget-wrapper[6228]: Traceback (most recent call last): Dec 27 08:07:42 dom0 widget-wrapper[6228]: File "/usr/lib64/python3.11/threading.py", line 1045, in _bootstrap_inner Dec 27 08:07:42 dom0 widget-wrapper[6228]: self.run() Dec 27 08:07:42 dom0 widget-wrapper[6228]: File "/usr/lib64/python3.11/threading.py", line 982, in run Dec 27 08:07:42 dom0 widget-wrapper[6228]: self._target(*self._args, **self._kwargs) Dec 27 08:07:42 dom0 widget-wrapper[6228]: File "/usr/lib/python3.11/site-packages/qui/updater/summary_page.py", line 298, in perform_restart Dec 27 08:07:42 dom0 widget-wrapper[6228]: self.restart_vms(to_restart) Dec 27 08:07:42 dom0 widget-wrapper[6228]: File "/usr/lib/python3.11/site-packages/qui/updater/summary_page.py", line 334, in restart_vms Dec 27 08:07:42 dom0 widget-wrapper[6228]: vm.start() Dec 27 08:07:42 dom0 widget-wrapper[6228]: File "/usr/lib/python3.11/site-packages/qubesadmin/vm/__init__.py", line 103, in start Dec 27 08:07:42 dom0 widget-wrapper[6228]: self.qubesd_call(self._method_dest, 'admin.vm.Start') Dec 27 08:07:42 dom0 widget-wrapper[6228]: File "/usr/lib/python3.11/site-packages/qubesadmin/base.py", line 76, in qubesd_call Dec 27 08:07:42 dom0 widget-wrapper[6228]: return self.app.qubesd_call(dest, method, arg, payload, Dec 27 08:07:42 dom0 widget-wrapper[6228]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Dec 27 08:07:42 dom0 widget-wrapper[6228]: File "/usr/lib/python3.11/site-packages/qubesadmin/app.py", line 770, in qubesd_call Dec 27 08:07:42 dom0 widget-wrapper[6228]: return self._parse_qubesd_response(return_data) Dec 27 08:07:42 dom0 widget-wrapper[6228]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Dec 27 08:07:42 dom0 widget-wrapper[6228]: File "/usr/lib/python3.11/site-packages/qubesadmin/base.py", line 111, in _parse_qubesd_response Dec 27 08:07:42 dom0 widget-wrapper[6228]: raise exc_class(format_string, *args) Dec 27 08:07:42 dom0 widget-wrapper[6228]: qubesadmin.exc.QubesException: Start failed: internal error: libxenlight failed to create new domain 'sys-net', see /var/log/libvirt/libxl/libxl-driver.> ``` #### `/var/log/libvirt/libxl/libxl-driver.log` ``` 2023-12-27 07:51:39.983+0000: libxl: libxl_pci.c:1585:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:00:0d.0 2023-12-27 07:51:39.988+0000: libxl: libxl_pci.c:1585:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:00:0d.2 2023-12-27 07:51:39.991+0000: libxl: libxl_pci.c:1585:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:00:14.0 2023-12-27 08:07:13.671+0000: libxl: libxl_pci.c:1585:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:00:0d.0 2023-12-27 08:07:13.806+0000: libxl: libxl_pci.c:1585:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:00:0d.2 2023-12-27 08:07:13.915+0000: libxl: libxl_pci.c:2096:do_pci_remove: Domain 2:xc_physdev_unmap_pirq irq=16: Invalid argument 2023-12-27 08:07:13.915+0000: libxl: libxl_pci.c:1585:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:00:14.0 2023-12-27 08:07:20.494+0000: libxl: libxl_qmp.c:1837:qmp_ev_parse_error_messages: Domain 15:Could not open '/sys/bus/pci/devices/0000:00:00.0/config': No such file or directory 2023-12-27 08:07:20.494+0000: libxl: libxl_pci.c:1273:pci_add_qmp_device_add_cb: Domain 15:Retrying PCI add 1 2023-12-27 08:07:21.561+0000: libxl: libxl_qmp.c:1837:qmp_ev_parse_error_messages: Domain 15:Could not open '/sys/bus/pci/devices/0000:00:00.0/config': No such file or directory 2023-12-27 08:07:21.562+0000: libxl: libxl_pci.c:1273:pci_add_qmp_device_add_cb: Domain 15:Retrying PCI add 2 2023-12-27 08:07:22.629+0000: libxl: libxl_qmp.c:1837:qmp_ev_parse_error_messages: Domain 15:Could not open '/sys/bus/pci/devices/0000:00:00.0/config': No such file or directory 2023-12-27 08:07:22.630+0000: libxl: libxl_pci.c:1273:pci_add_qmp_device_add_cb: Domain 15:Retrying PCI add 3 2023-12-27 08:07:23.697+0000: libxl: libxl_qmp.c:1837:qmp_ev_parse_error_messages: Domain 15:Could not open '/sys/bus/pci/devices/0000:00:00.0/config': No such file or directory 2023-12-27 08:07:23.699+0000: libxl: libxl_pci.c:1273:pci_add_qmp_device_add_cb: Domain 15:Retrying PCI add 4 2023-12-27 08:07:24.765+0000: libxl: libxl_qmp.c:1837:qmp_ev_parse_error_messages: Domain 15:Could not open '/sys/bus/pci/devices/0000:00:00.0/config': No such file or directory 2023-12-27 08:07:24.766+0000: libxl: libxl_pci.c:1273:pci_add_qmp_device_add_cb: Domain 15:Retrying PCI add 5 2023-12-27 08:07:25.833+0000: libxl: libxl_qmp.c:1837:qmp_ev_parse_error_messages: Domain 15:Could not open '/sys/bus/pci/devices/0000:00:00.0/config': No such file or directory 2023-12-27 08:07:25.833+0000: libxl: libxl_pci.c:1273:pci_add_qmp_device_add_cb: Domain 15:Retrying PCI add 6 2023-12-27 08:07:26.897+0000: libxl: libxl_qmp.c:1837:qmp_ev_parse_error_messages: Domain 15:Could not open '/sys/bus/pci/devices/0000:00:00.0/config': No such file or directory 2023-12-27 08:07:26.898+0000: libxl: libxl_pci.c:1273:pci_add_qmp_device_add_cb: Domain 15:Retrying PCI add 7 2023-12-27 08:07:27.965+0000: libxl: libxl_qmp.c:1837:qmp_ev_parse_error_messages: Domain 15:Could not open '/sys/bus/pci/devices/0000:00:00.0/config': No such file or directory 2023-12-27 08:07:27.966+0000: libxl: libxl_pci.c:1273:pci_add_qmp_device_add_cb: Domain 15:Retrying PCI add 8 2023-12-27 08:07:29.033+0000: libxl: libxl_qmp.c:1837:qmp_ev_parse_error_messages: Domain 15:Could not open '/sys/bus/pci/devices/0000:00:00.0/config': No such file or directory 2023-12-27 08:07:29.034+0000: libxl: libxl_pci.c:1273:pci_add_qmp_device_add_cb: Domain 15:Retrying PCI add 9 2023-12-27 08:07:30.097+0000: libxl: libxl_qmp.c:1837:qmp_ev_parse_error_messages: Domain 15:Could not open '/sys/bus/pci/devices/0000:00:00.0/config': No such file or directory 2023-12-27 08:07:30.098+0000: libxl: libxl_pci.c:1273:pci_add_qmp_device_add_cb: Domain 15:Retrying PCI add 10 2023-12-27 08:07:30.403+0000: libxl: libxl_pci.c:1862:device_pci_add_done: Domain 15:libxl__device_pci_add failed for PCI device 0:a5:0.0 (rc -9) 2023-12-27 08:07:30.403+0000: libxl: libxl_create.c:2000:domcreate_attach_devices: Domain 15:unable to add pci devices 2023-12-27 08:07:40.576+0000: libxl: libxl_device.c:1489:libxl__wait_for_backend: Backend /local/domain/0/backend/pci/16/0 not ready ```

otter2 · September 25, 2024, 8:37am

`guest-sys-usb-dm.log`

There is a lot in here, looking into startup attempt at 05:31…

By now, I have found this:

[2024-09-25 05:00:32] xen:swiotlb_xen: Warning: only able to allocate 4 MB for software IO TLB
[2024-09-25 05:00:32] xen:swiotlb_xen: Lowering to 2MB
[2024-09-25 05:00:32] xen:swiotlb_xen: Lowering to 2MB
[2024-09-25 05:00:32] xen:swiotlb_xen: Lowering to 2MB
[2024-09-25 05:00:32] xen:swiotlb_xen: Failed to get contiguous memory for DMA from Xen!
[2024-09-25 05:00:32] You either: don't have the permissions, do not have enough free memory under 4GB, or the hypervisor memory is too fragmented! (rc:-12)
[2024-09-25 05:00:32] pcifront pci-0: Could not setup SWIOTLB!
[2024-09-25 05:00:32] pcifront pci-0: 12 Error setting up PCI Frontend
[2024-09-25 05:00:32] blkfront: xvda: flush diskcache: enabled; persistent grants: enabled; indirect descriptors: enabled; bounce buffer: enabled
[2024-09-25 05:00:32]  xvda: xvda1 xvda2 xvda3
[2024-09-25 05:00:32] pcifront pci-0: backend going away!

`guest-sys-usb.log`

guest-sys-usb.log has pretty much nothing in it:

[2024-09-25 04:59:01] Logfile Opened
[2024-09-25 05:00:31] Logfile Opened
[2024-09-25 05:01:53] Logfile Opened
[2024-09-25 05:05:20] Logfile Opened
[2024-09-25 05:05:24] [    0.000000] Linux version 6.6.48-1.qubes.fc37.x86_64 (mockbuild@065a31b3c1ba4c34ab3938416488814f) (gcc (GCC) 12.3.1 20230508 (Red Hat 12.3.1-1), GNU ld version 2.38-27.fc37) #1 SMP PREEMPT_DYNAMIC Wed Sep  4 01:09:59 GMT 2024

I think it have started correctly at 05:05:20, so everything below should be irrelevant.

Github issues

unman’s issue seems to be somewhat similar, and yet they can start the qube manually. Maybe it’s due to differences in hardware.

icequbes1 has logs in their issue, but I cannot find my version of Assigning real physical device 04:00.0 at all. It just fails to get contiguous memory for DMA from Xen!

I am going to read guest-sys-usb-dm.log more later, and maybe add more information to unman’s issue.

apparatus · September 25, 2024, 8:52am

otter2:

[2024-09-25 05:00:32] xen:swiotlb_xen: Warning: only able to allocate 4 MB for software IO TLB
[2024-09-25 05:00:32] xen:swiotlb_xen: Lowering to 2MB
[2024-09-25 05:00:32] xen:swiotlb_xen: Lowering to 2MB
[2024-09-25 05:00:32] xen:swiotlb_xen: Lowering to 2MB
[2024-09-25 05:00:32] xen:swiotlb_xen: Failed to get contiguous memory for DMA from Xen!
[2024-09-25 05:00:32] You either: don't have the permissions, do not have enough free memory under 4GB, or the hypervisor memory is too fragmented! (rc:-12)
[2024-09-25 05:00:32] pcifront pci-0: Could not setup SWIOTLB!
[2024-09-25 05:00:32] pcifront pci-0: 12 Error setting up PCI Frontend
[2024-09-25 05:00:32] blkfront: xvda: flush diskcache: enabled; persistent grants: enabled; indirect descriptors: enabled; bounce buffer: enabled
[2024-09-25 05:00:32]  xvda: xvda1 xvda2 xvda3
[2024-09-25 05:00:32] pcifront pci-0: backend going away!

Try to set swiotlb=8192 kernel option for sys-usb:

github.com/QubesOS/qubes-issues

increase 'DMA buffer size' to 'swiotlb=8192' by default

opened 06:09PM - 26 Jun 15 UTC

closed 05:05AM - 01 Jul 15 UTC

adrelanos

As per [WiFi Realtek RTL8191SEvB Issue and Solution](https://groups.google.com/d…/msg/qubes-users/kMGTSwP72aU/BRvTJEbkGBQJ), I suggest increase `DMA buffer size` to `swiotlb=8192` by default. Unless something speaks against that? Maybe a duplicate of #365? https://github.com/QubesOS/qubes-issues/issues/365#issuecomment-77755780 > This is really a Xen-specific thing, not ours, closing. This may be true, but I would argue, that this is a grave usability issue for new users and therefore also Qubes responsibility to deliver a functioning product. So... - Has this issue been reported upstream? - What about increasing the default?

otter2 · November 4, 2024, 9:36am

Sorry for being pretty sluggish, I constantly forget about this issue.

Upon inspection, sys-usb doesn’t have any kernel options set (just as sys-net, it seems like all HVM sys-qubes don’t have it).

Nonetheless, I tried it with 2048 and 4096 (because I forgot that I should set it to 8192) and it doesn’t work when it’s already stuck. I’ll monitor the behaviour for new occasions and try 8192 if it fails again.

EDIT: Setting swiotlb seems to prevent failures, but cannot recover if already stuck. Still inconclusive, needs more testing