To confirm this interpretation, can anyone with an Intel post-Meteor Lake processor tell us if there is more than just one PCIe Root Complex on their device? A lspci -tvnn output shows it.
Bertrand
lspci -tvnn
-+-[0000:00]-+-00.0 Intel Corporation Device [8086:7d2d]
| +-01.0-[01]----00.0 Nextorage NN4LE NVMe SSD (DRAM-less) [1f31:451b]
| +-02.0 Intel Corporation Arrow Lake-S [Intel Graphics] [8086:7d67]
| +-04.0 Intel Corporation Device [8086:ad03]
| +-06.0-[02]--+-00.0 NVIDIA Corporation GB205M [GeForce RTX 5070 Ti Mobile] [10de:2f58]
| | \-00.1 NVIDIA Corporation GB205 High Definition Audio Controller [10de:2f80]
| +-06.1-[03]----00.0 Intel Corporation Wi-Fi 7(802.11be) AX1775*/AX1790*/BE20*/BE401/BE1750* 2x2 [8086:272b]
| +-07.0-[04-18]--
| +-07.1-[19-2d]--
| +-08.0 Intel Corporation Device [8086:ae4c]
| +-0a.0 Intel Corporation Device [8086:ad0d]
| +-0b.0 Intel Corporation Arrow Lake NPU [8086:ad1d]
| +-0d.0 Intel Corporation Meteor Lake-P Thunderbolt 4 USB Controller [8086:7ec0]
| +-0d.2 Intel Corporation Meteor Lake-P Thunderbolt 4 NHI #0 [8086:7ec2]
| +-14.0 Intel Corporation Device [8086:ae7f]
| +-1f.0 Intel Corporation Device [8086:ae10]
| \-1f.5 Intel Corporation Device [8086:ae23]
\-[0000:80]-+-14.0 Intel Corporation Device [8086:7f6e]
+-14.5 Intel Corporation Device [8086:7f2f]
+-16.0 Intel Corporation Device [8086:7f68]
+-19.0 Intel Corporation Device [8086:7f7a]
+-19.1 Intel Corporation Device [8086:7f7b]
+-1c.0-[81]----00.0 Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller [10ec:8125]
+-1e.0 Intel Corporation Device [8086:7f28]
+-1f.0 Intel Corporation Device [8086:7f0c]
+-1f.3 Intel Corporation Device [8086:7f50]
+-1f.4 Intel Corporation Device [8086:7f23]
\-1f.5 Intel Corporation Device [8086:7f24]
I’m thinking of doing the libvirt patch since I can’t use sysusb since my internal keyboard is attached to 80.14.0 and other weird usb stuff is happening with passthrough.
I’ll also add that usb-c displays and thunderbolt are working with the following options. Note unplugging active display or thunderbolt will probably crash your system–requiring a hard reboot.
# Disable authorized_default since thunderbolt needs authorization
#GRUB_CMDLINE_LINUX="$GRUB_CMDLINE_LINUX usbcore.authorized_default=0"
#Enable pci hotplug since required for thunderbolt
GRUB_CMDLINE_LINUX="$GRUB_CMDLINE_LINUX CONFIG_HOTPLUG_PCI=y CONFIG_HOTPLUG_PCI_PCIE=y"
Thank you so much for this output. So yes, you have two PCIe Root complex. What laptop is it?
I understand the sys-usb PCIe passthrough does not work well. So the patch may help you. Once you have tried it, can you please check the qvm-pci command to see if you have the same behaviour as mine?
If you want better texts describing your hardware, you can use sudo update-pciids in a VM and copy the updated /usr/share/hwdata/pci.ids file in your dom0 at the same place. If you want qvm-pci updated too, just reboot.
Bertrand
qvm-pci
dom0:00_00.0 PCI_Bridge: Intel Corporation Arrow Lake-HX 8p+16e cores Host Bridge
dom0:00_02.0 Display: Intel Corporation Arrow Lake-S [Intel Graphics]
dom0:00_04.0 Signal processing controller: Intel Corporation unknown pci device
dom0:00_06.0 PCI_Bridge: Intel Corporation Arrow Lake-HX PCIe Root Port #13
dom0:00_06.0-00_00.0 Storage: Samsung Electronics Co Ltd NVMe SSD Controller PM9C1a (DRAM-less)
dom0:00_06.3 PCI_Bridge: Intel Corporation unknown pci device
dom0:00_06.3-00_00.0 Display: NVIDIA Corporation GB205M [GeForce RTX 5070 Ti Mobile]
dom0:00_06.3-00_00.1 Audio: NVIDIA Corporation GB205 High Definition Audio Controller sys-audio (attached)
dom0:00_07.0 PCI_Bridge: Intel Corporation Meteor Lake-P Thunderbolt 4 PCI Express Root Port #0
dom0:00_07.1 PCI_Bridge: Intel Corporation Meteor Lake-P Thunderbolt 4 PCI Express Root Port #1
dom0:00_08.0 System peripheral: Intel Corporation Arrow Lake-HX Gauss Newton Algorithm (GNA)
dom0:00_0a.0 Signal processing controller: Intel Corporation Arrow Lake-HX Crash Log & Telemetry
dom0:00_0b.0 Processing accelerators: Intel Corporation Arrow Lake NPU
dom0:00_0d.0 PCI_USB: Intel Corporation Meteor Lake-P Thunderbolt 4 USB Controller sys-usb (attached: no-strict-reset=True)
dom0:00_0d.2 PCI_USB: Intel Corporation Meteor Lake-P Thunderbolt 4 NHI #0 sys-usb (attached: no-strict-reset=True)
dom0:00_14.0 Memory: Intel Corporation Arrow Lake-HX Shared SRAM (SOC-S)
dom0:00_1f.0 PCI_Bridge: Intel Corporation Arrow Lake-HX Direct eSPI Controller
dom0:00_1f.5 PCI_Serial_Bus: Intel Corporation Arrow Lake-HX SPI (flash) Controller
dom0:80_14.0 PCI_USB: Intel Corporation 800 Series PCH USB 3.1 xHCI HC sys-usb (attached)
dom0:80_14.3 Network: Intel Corporation Arrow Lake-S PCH CNVi WiFi sys-net (attached)
dom0:80_14.5 Non-VGA unclassified device: Intel Corporation unknown pci device
dom0:80_15.0 PCI_Serial_Bus: Intel Corporation 800 Series PCH I2C Controller #0
dom0:80_15.1 PCI_Serial_Bus: Intel Corporation unknown pci device
dom0:80_15.2 PCI_Serial_Bus: Intel Corporation unknown pci device
dom0:80_15.3 PCI_Serial_Bus: Intel Corporation 800 Series PCH I2C Controller #3
dom0:80_16.0 Communication controller: Intel Corporation 800 Series PCH HECI #1
dom0:80_19.0 PCI_Serial_Bus: Intel Corporation 800 Series PCH I2C Controller #4
dom0:80_19.1 PCI_Serial_Bus: Intel Corporation 800 Series PCH I2C Controller #5
dom0:80_1c.0 PCI_Bridge: Intel Corporation unknown pci device
dom0:80_1c.0-00_00.0 SD Host controller: Genesys Logic, Inc GL9767 PCIe SD UHS-II & SD Express Card Reader Controller
dom0:80_1c.6 PCI_Bridge: Intel Corporation 800 Series PCH PCIe Root Port #7
dom0:80_1c.6-00_00.0 Network: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller sys-net (attached)
dom0:80_1e.0 Communication controller: Intel Corporation unknown pci device
dom0:80_1e.3 PCI_Serial_Bus: Intel Corporation unknown pci device
dom0:80_1f.0 PCI_Bridge: Intel Corporation HM870 Chipset LPC/eSPI Controller
dom0:80_1f.3 Audio: Intel Corporation 800 Series ACE (Audio Context Engine) sys-audio (attached: no-strict-reset=True)
dom0:80_1f.4 PCI_Serial_Bus: Intel Corporation 800 Series PCH SMBus Controller
dom0:80_1f.5 PCI_Serial_Bus: Intel Corporation 800 Series PCH SPI (flash) Controller
Regards, Bloged
Thank you so much.
Now, I have to find what are the differences…
user@dom0:/sys/devices$ sudo qvm-pci
BACKEND:DEVID DESCRIPTION USED BY
dom0:00_00.0 PCI_Bridge: Intel Corporation Arrow Lake-HX 8p+16e cores Host Bridge
dom0:00_01.0 PCI_Bridge: Intel Corporation Meteor Lake-H PCIe Root Port #12
dom0:00_01.0-00_00.0 Storage: Samsung Electronics Co Ltd NVMe SSD 9100 PRO [PM9E1]
dom0:00_02.0 Display: Intel Corporation Arrow Lake-S [Intel Graphics]
dom0:00_04.0 Signal processing controller: Intel Corporation unknown pci device
dom0:00_06.0 PCI_Bridge: Intel Corporation Arrow Lake-HX PCIe Root Port #13
dom0:00_06.0-00_00.0 Display: NVIDIA Corporation GB207GLM [RTX PRO 1000 Blackwell Generation Laptop GPU]
dom0:00_06.0-00_00.1 Audio: NVIDIA Corporation GB207 High Definition Audio Controller
dom0:00_07.0 PCI_Bridge: Intel Corporation Meteor Lake-P Thunderbolt 4 PCI Express Root Port #0
dom0:00_08.0 System peripheral: Intel Corporation Arrow Lake-HX Gauss Newton Algorithm (GNA)
dom0:00_0a.0 Signal processing controller: Intel Corporation Arrow Lake-HX Crash Log & Telemetry
dom0:00_0b.0 Processing accelerators: Intel Corporation Arrow Lake NPU
dom0:00_0d.0 PCI_USB: Intel Corporation Meteor Lake-P Thunderbolt 4 USB Controller
dom0:00_0d.2 PCI_USB: Intel Corporation Meteor Lake-P Thunderbolt 4 NHI #0
dom0:00_14.0 Memory: Intel Corporation Arrow Lake-HX Shared SRAM (SOC-S)
dom0:00_1f.0 PCI_Bridge: Intel Corporation Arrow Lake-HX Direct eSPI Controller
dom0:00_1f.5 PCI_Serial_Bus: Intel Corporation Arrow Lake-HX SPI (flash) Controller
dom0:80_14.0 PCI_USB: Intel Corporation 800 Series PCH USB 3.1 xHCI HC
dom0:80_14.5 Non-VGA unclassified device: Intel Corporation unknown pci device
dom0:80_15.0 PCI_Serial_Bus: Intel Corporation 800 Series PCH I2C Controller #0
dom0:80_15.3 PCI_Serial_Bus: Intel Corporation 800 Series PCH I2C Controller #3
dom0:80_16.0 Communication controller: Intel Corporation 800 Series PCH HECI #1
dom0:80_19.0 PCI_Serial_Bus: Intel Corporation 800 Series PCH I2C Controller #4
dom0:80_19.1 PCI_Serial_Bus: Intel Corporation 800 Series PCH I2C Controller #5
dom0:80_1b.0 PCI_Bridge: Intel Corporation 800 Series PCH PCIe Root Port #17
dom0:80_1b.4 PCI_Bridge: Intel Corporation 800 Series PCH PCIe Root Port #21
dom0:80_1b.4-00_00.0 PCI_Bridge: Intel Corporation JHL9580 Thunderbolt 5 80/120G Bridge [Barlow Ridge Host 80G 2023]
dom0:80_1b.4-00_00.0-00_00.0 PCI_Bridge: Intel Corporation JHL9580 Thunderbolt 5 80/120G Bridge [Barlow Ridge Host 80G 2023]
dom0:80_1b.4-00_00.0-00_00.0-00_00.0 PCI_USB: Intel Corporation JHL9580 Thunderbolt 5 80/120G NHI [Barlow Ridge Host 80G 2023]
dom0:80_1b.4-00_00.0-00_01.0 PCI_Bridge: Intel Corporation JHL9580 Thunderbolt 5 80/120G Bridge [Barlow Ridge Host 80G 2023]
dom0:80_1b.4-00_00.0-00_01.0-00_00.0 PCI_Bridge: Intel Corporation JHL9480 Thunderbolt 5 80/120G Bridge [Barlow Ridge Hub 80G 2023]
dom0:80_1b.4-00_00.0-00_01.0-00_00.0-00_00.0 PCI_Bridge: Intel Corporation JHL9480 Thunderbolt 5 80/120G Bridge [Barlow Ridge Hub 80G 2023]
dom0:80_1b.4-00_00.0-00_01.0-00_00.0-00_01.0 PCI_Bridge: Intel Corporation JHL9480 Thunderbolt 5 80/120G Bridge [Barlow Ridge Hub 80G 2023]
dom0:80_1b.4-00_00.0-00_01.0-00_00.0-00_02.0 PCI_Bridge: Intel Corporation JHL9480 Thunderbolt 5 80/120G Bridge [Barlow Ridge Hub 80G 2023]
dom0:80_1b.4-00_00.0-00_01.0-00_00.0-00_03.0 PCI_Bridge: Intel Corporation JHL9480 Thunderbolt 5 80/120G Bridge [Barlow Ridge Hub 80G 2023]
dom0:80_1b.4-00_00.0-00_02.0 PCI_Bridge: Intel Corporation JHL9580 Thunderbolt 5 80/120G Bridge [Barlow Ridge Host 80G 2023]
dom0:80_1b.4-00_00.0-00_02.0-00_00.0 PCI_USB: Intel Corporation JHL9580 Thunderbolt 5 80/120G USB Controller [Barlow Ridge Host 80G 2023]
dom0:80_1b.4-00_00.0-00_03.0 PCI_Bridge: Intel Corporation JHL9580 Thunderbolt 5 80/120G Bridge [Barlow Ridge Host 80G 2023]
dom0:80_1c.0 PCI_Bridge: Intel Corporation 800 Series PCH PCIe Root Port #1
dom0:80_1c.6 PCI_Bridge: Intel Corporation 800 Series PCH PCIe Root Port #7
dom0:80_1c.6-00_00.0 Network: Intel Corporation Ethernet Controller I226-V sys-net (attached)
dom0:80_1d.0 PCI_Bridge: Intel Corporation 800 Series PCH PCIe Root Port #9
dom0:80_1d.4 PCI_Bridge: Intel Corporation 800 Series PCH PCIe Root Port #13
dom0:80_1d.4-00_00.0 SD Host controller: O2 Micro, Inc. unknown pci device
dom0:80_1d.6 PCI_Bridge: Intel Corporation 800 Series PCH PCIe Root Port #15
dom0:80_1d.6-00_00.0 Network: Intel Corporation Wi-Fi 7(802.11be) AX1775*/AX1790*/BE20*/BE401/BE1750* 2x2 sys-net (attached)
dom0:80_1f.0 PCI_Bridge: Intel Corporation unknown pci device
dom0:80_1f.3 Audio: Intel Corporation 800 Series ACE (Audio Context Engine)
dom0:80_1f.4 PCI_Serial_Bus: Intel Corporation 800 Series PCH SMBus Controller
dom0:80_1f.5 PCI_Serial_Bus: Intel Corporation 800 Series PCH SPI (flash) Controller
Here is mine. This is before i launch my sys-usb. I have a TB5 docking station connected.
Bertrand
Why do you have an Meteor Lake-H PCI-e Root port?
These are my root ports:
dom0:00_06.0 PCI_Bridge: Intel Corporation Arrow Lake-HX PCIe Root Port #13
dom0:00_07.0 PCI_Bridge: Intel Corporation Meteor Lake-P Thunderbolt 4 PCI Express Root Port #0
dom0:00_07.1 PCI_Bridge: Intel Corporation Meteor Lake-P Thunderbolt 4 PCI Express Root Port #1
dom0:80_1c.6 PCI_Bridge: Intel Corporation 800 Series PCH PCIe Root Port #7
And these are the ones mentioning Meteor:
dom0:00_07.0 PCI_Bridge: Intel Corporation Meteor Lake-P Thunderbolt 4 PCI Express Root Port #0
dom0:00_07.1 PCI_Bridge: Intel Corporation Meteor Lake-P Thunderbolt 4 PCI Express Root Port #1
dom0:00_0d.0 PCI_USB: Intel Corporation Meteor Lake-P Thunderbolt 4 USB Controller sys-usb (attached: no-strict-reset=True)
dom0:00_0d.2 PCI_USB: Intel Corporation Meteor Lake-P Thunderbolt 4 NHI #0 sys-usb (attached: no-strict-reset=True)
Regards, Bloged
I think the reason is that my processor (Intel Core Ultra 9 275HX) uses for some PCIe/PCI devices the same IDs than Meteor Lake ones.
My root ports:
00:01.0 PCI bridge [0604]: Intel Corporation Meteor Lake-H PCIe Root Port #12 [8086:7ecc] (rev 10)
00:06.0 PCI bridge [0604]: Intel Corporation Arrow Lake-HX PCIe Root Port #13 [8086:ae4d] (rev 10)
00:07.0 PCI bridge [0604]: Intel Corporation Meteor Lake-P Thunderbolt 4 PCI Express Root Port #0 [8086:7ec4] (rev 02)
80:1b.0 PCI bridge [0604]: Intel Corporation 800 Series PCH PCIe Root Port #17 [8086:7f40] (rev 10)
80:1b.4 PCI bridge [0604]: Intel Corporation 800 Series PCH PCIe Root Port #21 [8086:7f44] (rev 10)
80:1c.0 PCI bridge [0604]: Intel Corporation 800 Series PCH PCIe Root Port #1 [8086:7f38] (rev 10)
80:1c.6 PCI bridge [0604]: Intel Corporation 800 Series PCH PCIe Root Port #7 [8086:7f3e] (rev 10)
80:1d.0 PCI bridge [0604]: Intel Corporation 800 Series PCH PCIe Root Port #9 [8086:7f30] (rev 10)
80:1d.4 PCI bridge [0604]: Intel Corporation 800 Series PCH PCIe Root Port #13 [8086:7f34] (rev 10)
80:1d.6 PCI bridge [0604]: Intel Corporation 800 Series PCH PCIe Root Port #15 [8086:7f36] (rev 10)
The ones mentioning Meteor Lake:
00:01.0 PCI bridge [0604]: Intel Corporation Meteor Lake-H PCIe Root Port #12 [8086:7ecc] (rev 10)
00:07.0 PCI bridge [0604]: Intel Corporation Meteor Lake-P Thunderbolt 4 PCI Express Root Port #0 [8086:7ec4] (rev 02)
00:0d.0 USB controller [0c03]: Intel Corporation Meteor Lake-P Thunderbolt 4 USB Controller [8086:7ec0] (rev 02)
00:0d.2 USB controller [0c03]: Intel Corporation Meteor Lake-P Thunderbolt 4 NHI #0 [8086:7ec2] (rev 02)
I don’t have any device 00:07.1.
Bertrand
That is strange as I have the same processor.
It seems that both my USB-C ports are Thunderbolt enabled (which it shouldn’t according to the specs); that is probably why I have a dom0:00_07.1.
Regards, Bloged
Omen 16MAX. I spend a few hours working on getting the patch done. Turns out it only worked for me after fixing the libvirt.spec.in file from @Bloged
Patch0018: 0018-libxl-don-t-create-vkb-device-for-qubes-graphics-out.patch
Patch0019: 0019-arrow-lake-usb-fix.patch
Requires: libvirt-daemon-config-network = %{epoch}:%{version}-%{release}
%endif
Doesn’t work. Not good to only remove the opening IF statement and leave the closing IF statement.
Patch0018: 0018-libxl-don-t-create-vkb-device-for-qubes-graphics-out.patch
Patch0019: 0019-arrow-lake-usb-fix.patch
Requires: libvirt-daemon = %{epoch}:%{version}-%{release}
%if %{with_network}
Requires: libvirt-daemon-config-network = %{epoch}:%{version}-%{release}
%endif
Adding it back in allowed me to compile libvirt with Bloged githhub files otherwise unmodified–thank you @Bloged for the patch!
Thanks to @weird_pci for the build instructions.
I was able to start sysusb finally, though I still had to enable no-strict reset on 80.14.0 to make it work. Annoyingly, they put bluetooth on 80.14.0 and the internal keyboard on the same controller. I don’t have any trouble with qvm-pci, command outputs fine.
myuser@dom0:~$ qvm-pci
BACKEND:DEVID DESCRIPTION USED BY
dom0:00_00.0 PCI_Bridge: Intel Corporation unknown pci device
dom0:00_01.0 PCI_Bridge: Intel Corporation Meteor Lake-H PCIe Root Port #12
dom0:00_01.0-00_00.0 Storage: Nextorage NN4LE NVMe SSD (DRAM-less)
dom0:00_02.0 Display: Intel Corporation Arrow Lake-S [Intel Graphics]
dom0:00_04.0 Signal processing controller: Intel Corporation unknown pci device
dom0:00_06.0 PCI_Bridge: Intel Corporation unknown pci device
dom0:00_06.0-00_00.0 Display: NVIDIA Corporation GB205M [GeForce RTX 5070 Ti Mobile]
dom0:00_06.0-00_00.1 Audio: NVIDIA Corporation GB205 High Definition Audio Controller
dom0:00_06.1 PCI_Bridge: Intel Corporation Meteor Lake-H/U PCIe Root Port #10
dom0:00_06.1-00_00.0 Network: Intel Corporation Wi-Fi 7(802.11be) AX1775*/AX1790*/BE20*/BE401/BE1750* 2x2 sys-net (attached)
dom0:00_07.0 PCI_Bridge: Intel Corporation Meteor Lake-P Thunderbolt 4 PCI Express Root Port #0
dom0:00_07.1 PCI_Bridge: Intel Corporation Meteor Lake-P Thunderbolt 4 PCI Express Root Port #1
dom0:00_08.0 System peripheral: Intel Corporation unknown pci device
dom0:00_0a.0 Signal processing controller: Intel Corporation unknown pci device
dom0:00_0b.0 Processing accelerators: Intel Corporation Arrow Lake NPU
dom0:00_0d.0 PCI_USB: Intel Corporation Meteor Lake-P Thunderbolt 4 USB Controller
dom0:00_0d.2 PCI_USB: Intel Corporation Meteor Lake-P Thunderbolt 4 NHI #0
dom0:00_14.0 Memory: Intel Corporation unknown pci device
dom0:00_1f.0 PCI_Bridge: Intel Corporation unknown pci device
dom0:00_1f.5 PCI_Serial_Bus: Intel Corporation unknown pci device
dom0:80_14.0 PCI_USB: Intel Corporation unknown pci device sys-usb (attached)
dom0:80_14.5 Non-VGA unclassified device: Intel Corporation unknown pci device
dom0:80_16.0 Communication controller: Intel Corporation unknown pci device
dom0:80_19.0 PCI_Serial_Bus: Intel Corporation unknown pci device
dom0:80_19.1 PCI_Serial_Bus: Intel Corporation unknown pci device
dom0:80_1c.0 PCI_Bridge: Intel Corporation unknown pci device
dom0:80_1c.0-00_00.0 Network: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller sys-net (attached)
dom0:80_1e.0 Communication controller: Intel Corporation unknown pci device
dom0:80_1f.0 PCI_Bridge: Intel Corporation unknown pci device
dom0:80_1f.3 Audio: Intel Corporation unknown pci device
dom0:80_1f.4 PCI_Serial_Bus: Intel Corporation unknown pci device
dom0:80_1f.5 PCI_Serial_Bus: Intel Corporation unknown pci device
@bertrand
Well, I thought I didn’t have any trouble with qvm-pci. Turns out the commands prints the results fine, just without my thunderbolt pci devices! I’m unable to attach my thunderbolt pci devices to VMs. I can see them in lspci but not qvm-pci and not the VM gui for attaching. I’m going to try to revert to stock libvirt because thunderbolt not being passthrough-able is far worse for me than sys-usb not working. Looks like the command to revert is
sudo qubes-dom0-update --action=dsync libvirt-client
Hmm, even that and rebooting didn’t seem to restore qvm-pci. Looks liky dysnc didn’t work because it is looking at the cache.
Looks like the source of the qvm-pci problems may be a versioning issue. The builder versions are 10.5.0-2 wherease the original versions are 10.6.0-7.
I see no difference between the output of ls-pci and qvm-pci I can see all devices from lspci in the qvm-pci output.
If you update the version in the builder to 10.6.0 it should build that. So that everything is the same again. But the qubes-core-libvirt still points to 10.5.0 so where is 10.6.0 on your machine coming from?
Regards, Bloged
Looks like I was wrong and they are both 10.5.0-2. Even though I couldn’t use thunderbolt I think I will still try the patch again since I can use Oculink instead (yay for having two M.2 nvme slots). I also found I had the wrong address hidden from dom0 so . . . not entirely sure why qvm-pci had different results from lspci.
So now with the patch qvm-pci and lspci work fine for whatever reason. One gotcha is that the computer directs usb devices to different controllers based on the device type for the usb-c ports and maybe the others also. That means I get a different pci device if I plug in a usb 2.0 device than if I plug in a usb 3.0 device (i.e. uses 80.14.0 for usb 2.0 and 00:0d.0 for usb 3+). I have my displayport dock connected to a “downstream” port on my thunderbolt-pcie adapter which is then connected to the laptop.
Trying to run sys-usb with 00:0d.0 attached to sys-usb and then pass a displayport audio device to the vm tries to work but I only get audio connected even though the displays flash white for me. This is probably either a limitation in how qubes deals with displayport devices OR that the sysusb OS template is not new enough or doesn’t have appropriate drivers for the displayport device.
Once sysusb is run using the thunderbolt/usb3 port. That kills the thunderbolt pcie device passthrough because the something tried to use usb3 using the same port.
So anyway, external usb-c monitors do work in qubes with default authorization set to 1 in grub. External usb-c monitors can also be passed-thru to VMs by riding-along with a thunderbolt pcie device passthrough. Displayport may be passable to vms from sysusb if the bloged’s patch is applied and the sysusb template is new enough or has drivers installed–I may try this later with a different DP device or template adjustments.
Interestingly, I am able to start sysusb and attach my usb2 keyboard and mouse after connecting the displayport displays to the VM.
I still haven’t fully figured this out myself. Very confusing to say the least.
I’ve noticed you all have Nvidia on your laptops. Are you able to passthrough them to a VM?
I am on a desktop and GPU passthrough is important to me. I haven’t tried yet but I want to know if I should expect any issues.
-
I don’t think whether the device you input goes to to TB controller or the built in USB controller depends on the type, instead I think your Dock has different ports, some are TB ports (should have a special sign) and some are merely USB ports. Now, USB ports should be tunnelled directly and have the HOST usb port. But child TB ports are not, so if you put a USB device in those, the TB controller has to take care of it.
(not entirely sure). -
Did you remove your USB controller from Dom0? Perhaps you won’t need a FLRESET if you hide it from Dom0 via the qubes
rd.qubes.hide_pciparam in Grub?
(without it, your USB controller would start in Dom0 and stay there until sys-usb gets launched which will then unload the USB driver in dom0 and try to FLRESET, which as you know fails and instead a non-strict FlRESET is done). Checkingdmesgin dom0 will show if this is happening, you will see lines about the xhci driver being loaded and unloaded. -
Also, isn’t hotplug for Thunderbolt disabled in the Qubes kernel, meaning you can’t get PCI-E devices over Thundebolt?
Then there’s your Thunderbolt BIOS settings and also try boltctl in dom0 console.
I think it’s inherently insecure because PCI devices would always appear in Dom0. My understanding is that Qubes doesn’t help against a malicious PCI device. If you think about it, when you boot, your USB controller starts right away because your external keyboard works. This means, it’s not isolated at all and can DMA your RAM, ergo owns your PC. Then when Qubes is booted, the driver isn’t loaded so the PCI device is basically ignored but it can still mess things up if it wanted to. Finally when you start a domU(sys-usb), the PCI device is passthroughed, meaning its memory is isolated and has to pass through the IOMMU, so it can’t DM outside the domU memory
So allowing hotplug sounds like you’re risking the plugging of malicious PCI-E decices to instantly DMA your memory. Perhaps there’s some way to be done securely.
Btw I would be super happy to get corrections on the above. Feel free to correct and ammend my understanding.
Anyhow, I am still wondering about the best way to solve this IO thing, need to do more experiments with the patch and the TB ports I have, to figure out the best way.
Ideally I am thinking of using the patch simply because there’s a Bluetooth controller on the built-in USB, and ideally I want that inside sys-usb to use.
I have not been able to get the internal nvidia dgpu working in a vm. In fact, I decided to disable it entirely since it was making battery life horrible despite not being in use.
- Don’t take my word for it. Try it.
- Tried it.
- No, it works great. You just can’t hotplug it.
Thunderbolt works for me without bolt. I have no thunderbolt bios setting. The HP bios has almost no settings. Pretty sure qubes just trusts all connected devices that are not USB.
Regarding insecurity . . . sure. But I’m not going to give up thunderbolt.
It seems that GPU pass-trough isn’t a simple to test as I wished it was. If I have some time I am willing to test… when I have some spare time to tinker around… any good tutorials/guides to follow along?
Still haven’t got my external monitors working; that is a real bummer. No matter what I do I cannot get the USB-C/TB to work on my machine.
Kind regards, Bloged
1 Like
@Bloged Have you enabled DP-alt mode in bios if that is a setting?
@Bloged also, are you talking about usb-c monitors on dom0 or in a VM? It should work fine in dom0. Passing through a displayport dock with two monitors to a vm works for me if I passthrough the hidden usb controller (00:0d.0 same iommu group as thunderbolt controller) AND attach the displayport audio device from sys-usb (80.14.0).
@bertrand @Bloged
If I enroll my thunderbolt device (so it is automatically authorized) the qvm-pci consistently hangs. All this time I have had the thunderbolt controller hidden from dom0. In fact, I have to have it hidden or else the thunderbolt pci device (gpu) does not appear in qvm-pci and I cannot pass it to a vm!! Talk about counter-intuitive!