Major problem with qvm-pci and devices in general

User Support
1

hey guys, i am really new to qubes and i’ve been experimenting for a while now. everything seems to work quite fine, except for the touchpad that sometimes stops working and after 30 seconds it comes back again.

But i have encountered one big problem, while trying to create a new netVM based off of a template with the mac anonimizer configuration i saw that i cannot access the devices attached to each VM in the GUI, the devices tab is there but i cannot click it.
I tried attaching my net card to the vm using the terminal then, and the qvm-pci command doesn’t seem to work:

[user@dom0 ~] $ qvm-pci
Failed to list 'pci' devices, this device type either does not exist or you do not have access to it

this already wasn’t looking good. since qvm-pci wouldn’t work i tried lspci, wich turned out working, so i pulled the BDF of my card from there and tried attaching it using qvm-pci, which of course didn’t work:

qvm-pci attach net-macanon dom0:00_xx.x --persistent

quite a few lines of codes went through but the last one had an error that i think aborted the whole thing, sorry if i can’t provide all of the terminal lines but it seems i can’t copy to global clipboard from dom0 terminal, so i have to copy them all by hand. anyways here’s the line with the error:

File "/usr/lib/python3.8/site-packages/qubesadmin/base.py", line 87, in _parse_qubesd_response
  raise qubesadmin.exc.QubesDaemonAccessError(
qubesadmin.exc.QubesDaemonAccessError: Got empty response from qubesd. see journalctl in dom0 for details.

i checked journalctl as it said but i can’t really find anything, of course i also don’t know anything so you probably would find something, the problem is, as i said, that i can’ find a way to copy text from dom0 terminal to other VMs so i can’t show the journalctl text right now, i tried everyting but i can’t seem to find a way to copy and pasting the text.

All i can find on journalctl is one thing: AssertionError. it comes up everytim i try to do qvm-pci and presents itself like this:

File "/usr/lib/python3.8/site-packages/qubes/ext/pci.py", line 191, in on_device_list_pci
  yield PCIDevice(vm, None, libvirt_name=libvirt_name)
File "/usr/lib/python3.8/site-packages/qubes/ext/pci.py", line 140, in __init__
assert dev_match
AssertionError

this lines have some other lines behind them, they are all similar. it-s accessing some files in qubesadmin and when it gets to this specific file it just aborts everything. the whole thing in journalctl has this line before it:

unhandled exception while calling src=b'dom0' meth=b'admin.vm.device.pci.Available' dest=b'dom0' argb'' len(untrusted_payload)=0
traceback (most recent call last):
~and here starts the list of files that are accessed and that ends with the previously cited lines~

one thing i found was that this same error appeared when trying the attachment of my net card too. so it was the cause of the abort of both qvm-pci and the attempt of attachment of my net card. seems like qubes can’t read what devices are there, but it can read them using lspci… i’m really lost.

another thing i can say is that this problems have persisted through multiple installations, probably about 3. all of them done with a fresh iso of qubes.
if it can help i am on an asus f415 i think, the one with 11th gen i3.

i apologise for my probable errors in writing this post but i’m quite tired and english isn’t really my first language.

If anyone could help me i would be really really grateful, i can’t seem to find any info about this and i’m worried there probably isn’t anything to do and will probably be stuck without any mac spoofing for life ;_;
thank you in advance, i feel like i still forgot some info that would probably be helpful, but they don’t come to my mind, i’ll update later if they do. oh qubes version is 4.1 rc2. 4.0 installation wouldn’t even start so i had to stick with this.

Thanks

1 Like
2

found how to copy dom0 clipboard, it was actually pretty easy. so here are the various logs:
This is what happens in terminal when i try to attach the net card:

[user@dom0 ~]$ qvm-pci attach net-macanon dom0:00_14.3 --persistent
Traceback (most recent call last):
  File "/usr/bin/qvm-pci", line 5, in <module>
    sys.exit(main())
  File "/usr/lib/python3.8/site-packages/qubesadmin/tools/qvm_device.py", line 291, in main
    args = parser.parse_args(args, app=app)
  File "/usr/lib/python3.8/site-packages/qubesadmin/tools/__init__.py", line 409, in parse_args
    subaction.parse_qubes_app(self, namespace)
  File "/usr/lib/python3.8/site-packages/qubesadmin/tools/qvm_device.py", line 200, in parse_qubes_app
    dev = vm.devices[devclass][device_id]
  File "/usr/lib/python3.8/site-packages/qubesadmin/devices.py", line 282, in __getitem__
    for dev in self.available():
  File "/usr/lib/python3.8/site-packages/qubesadmin/devices.py", line 235, in available
    self._vm.qubesd_call(None,
  File "/usr/lib/python3.8/site-packages/qubesadmin/base.py", line 74, in qubesd_call
    return self.app.qubesd_call(dest, method, arg, payload,
  File "/usr/lib/python3.8/site-packages/qubesadmin/app.py", line 748, in qubesd_call
    return self._parse_qubesd_response(return_data)
  File "/usr/lib/python3.8/site-packages/qubesadmin/base.py", line 87, in _parse_qubesd_response
    raise qubesadmin.exc.QubesDaemonAccessError(
qubesadmin.exc.QubesDaemonAccessError: Got empty response from qubesd. See journalctl in dom0 for details.

it asks me to go to journalctl so here is what it looks like there:

Dec 27 11:55:07 dom0 qubesd[1822]: unhandled exception while calling src=b'dom0' meth=b'admin.vm.device.pci.Available' dest=b'dom0' arg=b'' len(untrusted_payload)=0
Dec 27 11:55:07 dom0 qubesd[1822]: Traceback (most recent call last):
Dec 27 11:55:07 dom0 qubesd[1822]:   File "/usr/lib/python3.8/site-packages/qubes/api/__init__.py", line 286, in respond
Dec 27 11:55:07 dom0 qubesd[1822]:     response = await self.mgmt.execute(
Dec 27 11:55:07 dom0 qubesd[1822]:   File "/usr/lib/python3.8/site-packages/qubes/api/admin.py", line 1217, in vm_device_available
Dec 27 11:55:07 dom0 qubesd[1822]:     devices = self.dest.devices[devclass].available()
Dec 27 11:55:07 dom0 qubesd[1822]:   File "/usr/lib/python3.8/site-packages/qubes/devices.py", line 376, in available
Dec 27 11:55:07 dom0 qubesd[1822]:     devices = self._vm.fire_event('device-list:' + self._bus)
Dec 27 11:55:07 dom0 qubesd[1822]:   File "/usr/lib/python3.8/site-packages/qubes/events.py", line 195, in fire_event
Dec 27 11:55:07 dom0 qubesd[1822]:     sync_effects, async_effects = self._fire_event(event, kwargs,
Dec 27 11:55:07 dom0 qubesd[1822]:   File "/usr/lib/python3.8/site-packages/qubes/events.py", line 168, in _fire_event
Dec 27 11:55:07 dom0 qubesd[1822]:     effects.extend(effect)
Dec 27 11:55:07 dom0 qubesd[1822]:   File "/usr/lib/python3.8/site-packages/qubes/ext/pci.py", line 191, in on_device_list_pci
Dec 27 11:55:07 dom0 qubesd[1822]:     yield PCIDevice(vm, None, libvirt_name=libvirt_name)
Dec 27 11:55:07 dom0 qubesd[1822]:   File "/usr/lib/python3.8/site-packages/qubes/ext/pci.py", line 140, in __init__
Dec 27 11:55:07 dom0 qubesd[1822]:     assert dev_match
Dec 27 11:55:07 dom0 qubesd[1822]: AssertionError

While here is what happens in the journal when i try and execute qvm-pci:

Dec 27 11:57:50 dom0 qubesd[1822]: unhandled exception while calling src=b'dom0' meth=b'admin.vm.device.pci.Available' dest=b'dom0' arg=b'' len(untrusted_payload)=0
Dec 27 11:57:50 dom0 qubesd[1822]: Traceback (most recent call last):
Dec 27 11:57:50 dom0 qubesd[1822]:   File "/usr/lib/python3.8/site-packages/qubes/api/__init__.py", line 286, in respond
Dec 27 11:57:50 dom0 qubesd[1822]:     response = await self.mgmt.execute(
Dec 27 11:57:50 dom0 qubesd[1822]:   File "/usr/lib/python3.8/site-packages/qubes/api/admin.py", line 1217, in vm_device_available
Dec 27 11:57:50 dom0 qubesd[1822]:     devices = self.dest.devices[devclass].available()
Dec 27 11:57:50 dom0 qubesd[1822]:   File "/usr/lib/python3.8/site-packages/qubes/devices.py", line 376, in available
Dec 27 11:57:50 dom0 qubesd[1822]:     devices = self._vm.fire_event('device-list:' + self._bus)
Dec 27 11:57:50 dom0 qubesd[1822]:   File "/usr/lib/python3.8/site-packages/qubes/events.py", line 195, in fire_event
Dec 27 11:57:50 dom0 qubesd[1822]:     sync_effects, async_effects = self._fire_event(event, kwargs,
Dec 27 11:57:50 dom0 qubesd[1822]:   File "/usr/lib/python3.8/site-packages/qubes/events.py", line 168, in _fire_event
Dec 27 11:57:50 dom0 qubesd[1822]:     effects.extend(effect)
Dec 27 11:57:50 dom0 qubesd[1822]:   File "/usr/lib/python3.8/site-packages/qubes/ext/pci.py", line 191, in on_device_list_pci
Dec 27 11:57:50 dom0 qubesd[1822]:     yield PCIDevice(vm, None, libvirt_name=libvirt_name)
Dec 27 11:57:50 dom0 qubesd[1822]:   File "/usr/lib/python3.8/site-packages/qubes/ext/pci.py", line 140, in __init__
Dec 27 11:57:50 dom0 qubesd[1822]:     assert dev_match
Dec 27 11:57:50 dom0 qubesd[1822]: AssertionError

as you can see, same thing.

The only piece of information i could find after a little digging was another post here on this forum about having the devices option in the GUI greyed out and not having access to qvm-pci, here’s the link [qubes-users] Qubes 4.1 qubes-manager "Devices" grayed out, but there aren’t any responses about the problem itself, the only response is about no-strict-reset which is another thing.

am i missing something?
thanks for the help, if anyone sees this.

3

Hello, created this acocunt to say that I also experienced the same issue a couple weeks back after a successful install of 4.1 rc2.

Device: MSI GE66 RAIDER 11UH

 CPU - Intel® Core™ i9-11980HK
 CHIPSET - HM570 
 GRAPHICS - NVIDIA® GeForce RTX™ 3080 Laptop GPU 16GB GDDR6
 NOTE: Attempted installation was on an m.2 ssd connected via USB-C

Details:
My attempts to install Qubes 4.1 rc2 are detailed below for possible clues starting from failing to install it to updating my bios and the install working properly and finally being faced with the error described in the OP.

1. Initial Attempts:

Initially the installation would not work at all on my device, it would get stuck with the typical EFI/Nvidia errors I cant recall exactly what they were other than getting stuck on a black screen after a few seconds but before the install.
I went ahead and adjusted the iso and for some reason there still was no change at all to the boot process. (Note: This is not my first Qubes installation and I have had to do similar adjustments to the boot options be it legacy or EFI mode for my previous lenovo device with an nvidia card) I managed to be able to pass boot options to it though through the EFI boot manager known as rEFInd. Next it seemed I ran into a kernel panic issue involving IO-APIC.

2. BIOS Update:

After looking around for a while for details about this I found that the BIOS for my device had recently been updated (mid Nov) and after updating the BIOS i tried again with a fresh 4.1 rc2 (unmodified) and it worked with no issue at all. Clear through on to the actual loading of the desktop everything worked flawlessly.

3. Current issue/OP issue:

Once loaded up, I began attempting to configure sys-net and ran into the same issue as the described in the OP. The device attachment tab in the VM properties menu (Qubes Manager>Right Click on a VM>Properties>Devices? tab) was greyed out and was not clickable. I went ahead and tried qvm-pci and the other methods described in the OP and reached the same results. This being said, the PCI devices are listed when using lspci through dom0. Whats more is that block devices are available when using qvm-devices. The qubes installation also knew I was using a usb external drive for the OS because it didn’t allow me to create a sys-usb.

Conclusion:
yes i am having the same error, I will try to install rc3 now and see if that fixes anything, also going to try to put the drive into the laptop rather than use it externally and see if that makes a difference.

monhibited: What are the specs of the system you are installing on?

4

hey man,
i’m on a low end laptop, cpu should be an i3 1115g4 with integrated graphics and i’m installing it on the built-in m2 ssd.

One thing that i noticed is that everyone having this issue is on 11th gen intel, could it really be the cause of this? i don’t know.
Seems there’s nothing to do anyways and hope for qubes devs to try and fix this issue while i do reinstall after reinstall…

5

I have the same issue, also with multiple tries to re-install.
I am on i7-11800H, so it is not only i9.

Has anyone found a way around?

RaspyVotan

6

To be sure it’s not a bug in qubes tools, have you tried attaching devices using Xen’s default commands, using xl, just to compare ?
You can see a quick explanation of the commands in this post, they are at the end under “Manual assignments”.
You can also check the Xen PCI Passthrough guide for more hints.
PS: xl commands need sudo/root.

7

A stupid question - is your qube HVM? I can’t seem to find that info in a correspondence.

8

Just tried the xl command. It works. However I’m not able to set the persistent flag and the no-strict-reset flag. I don’t know how that’s handled in qubes. So after a reboot of the VM, the pci device is no longer attached so this solution is useless.

sudo xl pci-list [vmname]
sudo xl pci-attach [vmname] 0000:00:14.0

It seems like this is a bug in the qubes admin tools.

9

Of course, xl pci-attach and detach are meant for HOTPLUGGING devices, so it’s not persistent by design. It’s also not the official way to do it in Qubes, it’s just a test.
You can try qvm-pci, which are the Qubes equivalents, and can set the persistent and no-strict-reset flags (man qvm-pci or man qvm-devices). I -think- the persistent flag is used when you attach devices from the Qubes manager.
Also, sometimes it helps to add the pciback thing in GRUB kernel cmd line. But I’m still waiting on an answer about the difference between Xen way (pciback.hide) and Qubes one (rd.qubes.hide).

BTW, the Xen way of attaching devices at boot time is by setting them in the domU config file. Qubes uses qubes.xml for global conf.
Another test would be to create a separate xml file for the domU, where you just add a block <devices class="pci">. The method is explained here : Custom libvirt (ie domU) config [! dirty doc - v4.1.25-0-ga7649998].
What’s good in your case is that PCI PT -is- working.

2 Likes
10

Now that was valuable information. Thank you.
I manually edited /var/lib/qubes/qubes.xml and everything works now. PCI devices are now persistently mapped with the reset option.

<devices class="pci">
  <device backend-domain="dom0" id="00_14.0">
    <option name="no-strict-reset">True</option>
  </device>
  ...
</devices>

Still, the Devices tab is greyed out in the qube properties (yes it’s an HVM qube).
Also, the commands qvm-devices or the alias qvm-pci are not working.

I hope this gives OP a clue to get a working system until this bug is fixed.

Thanks again!

11

Happy to help !
Take care that the qubes.xml file does not get overwritten, I think it’s built from the qubes manager. You may lose your settings.
That’s why I suggested to use the xen/by-name/DOMU.xml setting.
Maybe it’s greyed out cause you set a manual config and the manager is smart enough to not mess things up ? Or because the domU is running ?
That would be nice for devs to know why qvm-* commands don’t work, and maybe file a bug report. It could help others.

12

Hi, no the problem is there immediately after setup. I was struggling with setting up sys-net-as-usbvm, which also has a bug, but I understand that part will be hopefully rewritten in 4.2

Setup works fine and adds PCI devices the way it should be, as long as you don’t select the option to use sys-net as usb (which is required with new laptops that don’t have RJ45 ports). So what I did was create separate sys-net and sys-usb, detach the PCI devices from sys-usb and reattach them to sys-net. I had to do that all manually because the devices tab is greyed out from the very beginning.

I wish I could file a bug report, but that’s a bit out of my league. Hopefully the devs can pick this up. Happy to provide more info if needed.

13

And, creating sys-usb only, without sys-net, doesn’t provide network? I wonder how’s that, since by default all sys-usb’s are created to provide network, and pointing other qubes to it should work…

14

sys-usb only would only attach USB controller PCI devices. But yes, you would be right, I could have created sys-usb only and attach the right controllers. I used sys-net to check which PCI devices were connected and which features were enabled.

15

After a quick looking into the source code, I found this problem was caused by a failure during a regex matching, which expects every PCI device identifier starts with pci_0000.

But if I run sudo virsh nodedev-list in dom0, I found some of my PCI device identifier starts with pci_10000. This can also be verified using lspci.

I dirty-patched the source code to simply ignore any PCI device that don’t start with pci_0000, now the qvm-pci command starts to work and the device tab in Qube Settings becomes clickable, but both only list PCI devices that start with pci_0000.

It looks like Qubes doesn’t support multiple PCI domain for now, any idea on this?

Edit: I’m on XPS15 with i7-11800H.

16

I can confirm the validity of previous investigation reported by @ycdzj.

The root of the issue is that qubesd daemon fails parsing PCI devices on a system, which as a consequence makes all qubes (including system ones) fail to start, Devices tab in Qube Manager becomes disabled/grayed-out, and commands like e.g. qvm-pci give an error.

It fails to parse PCI devices because there are some PCI devices that don’t belong to PCI domain 0000.

Hardware in my case is Asus G614J. The issue can be confirmed by running lspci or virsh nodedev-list which in my case reports 3 PCI devices which do not belong to “0000” but to “10000”:

10000:e0:1d.0 System peripheral: Intel Coropration Device 09ab
10000:e0:1d.4 PCI bridge: Intel Corpration Device 7a34 (rev 11)
10000:e1:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd Device a80c

This breaks qubesd PCI-related functionality which expects or embeds a literal “0000” in a couple places.

So what happens is as follows: (the example is for running command qvm-pci, but it’s the same for Qubes manager etc.

  1. User runs qvm-pci
  2. This command goes to ask for “Available” PCI devices from qubesd. It does so for all domains (for each domain, including dom0, it asks for available devices)
  3. When it gets to domain dom0 where the offending PCI devices are, it triggers an exception (which is visible in journalctl -xef), returning empty response to the caller
  4. qvm-pci code then makes a further mistake by masking the real error text with a generic message: “Failed to list ‘pci’ devices, this device type either does not exist or you do not have access to it”

So to fix this ad-hoc, we can patch a line of Python code in dom0, and restart qubesd:

  1. Open qubes-core-admin/pci.py at master · QubesOS/qubes-core-admin · GitHub
  2. In function on_device_list_pci, above yield PCIDevice(...), add this code:
            if not libvirt_name.startswith("pci_0000_"):
              continue

Then restart qubesd with service qubesd restart.

Commands like qvm-pci and Qube Manager will start running. You might need to open Devices tab for system qubes and make sure that no “Unknown” devices are assigned to them (those will prevent qubes from starting, since they are invalid.)

And you will need to make sure not to assign any wrong/incorrect devices to a qube afterwards, as that might make your OS just keep rebooting on boot :slight_smile: (If by chance that happens, reboot in Grub press ‘e’ to edit the boot menu, find then line which mentions “vmlinuz”, and then at the end of that line add option “qubes.skip_autostart=1”. This will allow you to boot into the system and fix the issue.)

1 Like
17
1 Like