Issue with hiding GPU for passthrough

[privacyuser]

When I try hide my 4070 (want dom0 to run on the rx570 and have the 4070 for passthrough) what happens is I get this weird thing on my main monitor (plugged into displayport on the amd card, 2nd monitor plugged into nvidia hdmi)
My monitor plugged into my nvidia card loads, but the one plugged into the amd has this thing on it that stays the whole time the system is on

I have a NVIDIA 4070 and a amd rx 570. I successfully retrieved the IOMMU configuration -

00:02.0 Display controller [0380]: Intel Corporation AlderLake-S GT1 [8086:4680] (rev 0c)
IOMMU Group 1:
00:00.0 Host bridge [0600]: Intel Corporation 12th Gen Core Processor Host Bridge/DRAM Registers [8086:4660] (rev 02)
IOMMU Group 10:
00:1c.0 PCI bridge [0604]: Intel Corporation Device [8086:7a38] (rev 11)
IOMMU Group 11:
00:1c.2 PCI bridge [0604]: Intel Corporation Device [8086:7a3a] (rev 11)
IOMMU Group 12:
00:1c.4 PCI bridge [0604]: Intel Corporation Device [8086:7a3c] (rev 11)
IOMMU Group 13:
00:1f.0 ISA bridge [0601]: Intel Corporation Device [8086:7a04] (rev 11)
00:1f.3 Audio device [0403]: Intel Corporation Device [8086:7a50] (rev 11)
00:1f.4 SMBus [0c05]: Intel Corporation Device [8086:7a23] (rev 11)
00:1f.5 Serial bus controller [0c80]: Intel Corporation Device [8086:7a24] (rev 11)
IOMMU Group 14:
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation AD104 [GeForce RTX 4070] [10de:2786] (rev a1)
01:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:22bc] (rev a1)
IOMMU Group 15:
02:00.0 Non-Volatile memory controller [0108]: Sandisk Corp WD Black SN770 NVMe SSD [15b7:5017] (rev 01)
IOMMU Group 16:
04:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I225-V [8086:15f3] (rev 03)
IOMMU Group 17:
05:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] [1002:67df] (rev ef)
05:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590] [1002:aaf0]
IOMMU Group 2:
00:01.0 PCI bridge [0604]: Intel Corporation 12th Gen Core Processor PCI Express x16 Controller #1 [8086:460d] (rev 02)
IOMMU Group 3:
00:06.0 PCI bridge [0604]: Intel Corporation 12th Gen Core Processor PCI Express x4 Controller #0 [8086:464d] (rev 02)
IOMMU Group 4:
00:08.0 System peripheral [0880]: Intel Corporation 12th Gen Core Processor Gaussian & Neural Accelerator [8086:464f] (rev 02)
IOMMU Group 5:
00:0a.0 Signal processing controller [1180]: Intel Corporation Platform Monitoring Technology [8086:467d] (rev 01)
IOMMU Group 6:
00:14.0 USB controller [0c03]: Intel Corporation Device [8086:7a60] (rev 11)
00:14.2 RAM memory [0500]: Intel Corporation Device [8086:7a27] (rev 11)
IOMMU Group 7:
00:14.3 Network controller [0280]: Intel Corporation Device [8086:7a70] (rev 11)
IOMMU Group 8:
00:16.0 Communication controller [0780]: Intel Corporation Device [8086:7a68] (rev 11)
IOMMU Group 9:
00:17.0 SATA controller [0106]: Intel Corporation Device [8086:7a62] (rev 11)

Here is my grub configuration where I hide the nvidia card (and the audio device)
If I only hide the card it doesn’t make a difference

GRUB_CMDLINE_LINUX="rd.lvm.lv=qubes_dom0/root rd.lvm.lv=qubes_dom0/swap plymouth.ignore-serial-consoles 6.6.21-1.qubes.fc37.x86_64 x86_64 rhgb quiet splash nomodeset rd.qubes.hide_pci=01:00.0 rd.qubes.hide_pci=01:00.1 "

Did I screw up somewhere or is this a issue with my hardware?

Also - Does this post mean that qubes will support passthrough without weird configuration involved? Or does it just mean it will further isolate gui from dom0

Qubes gpu passthrough

GUI domain is under active development and will be shipped with 4.1 - you can test it with the alpha now.

[apparatus]

Does your rx570 with main monitor work correctly without hiding Nvidia card?

@privacyuser

rd.qubes.hide_pci=01:00.0 rd.qubes.hide_pci=01:00.1

I’m not sure what’s the correct syntax but try this:
rd.qubes.hide_pci=01:00.0,01:00.1

How are your GPUs connected to the motherboard?
Which one is connected to the first/top PCI slot?
Maybe there is an issue with Nvidia being Primary GPU and it’s somehow break things.
Check if you have some option in your BIOS to change the primary GPU.
Or try to switch GPU in PCI slots.

[privacyuser]

Motherboard has 2 slots. Nvidia card is in the first slot and the amd card is in the second. Qubes works properly on both gpu’s on my primary monitor. I’m fairly certain that the gpu pci slots aren’t the issue here (considering that the screen is mirrored up until I select which option I want in the boot loader. Trying out the syntax change and checking bios rn

[apparatus]

@privacyuser

I’m fairly certain that the gpu pci slots aren’t the issue here (considering that the screen is mirrored up until I select which option I want in the boot loader.

It could be an issue, also maybe there is some issue with cable type priority, you can try to disconnect the second monitor during boot:

GPU Passthrough -> lightdm.service won't start

It probably try to use the GPU you just hide from dom0. Assuming you used the correct value for rd.qubes.hide_pci: Only plug your primary GPU to your primary screen. Once you successfully booted into Qubes, plug the secondary GPU to the secondary screen Also did you checked my comment about this subject Note: if after this step when you reboot the computer you get stuck in the QubesOS startup that means you are trying to use the GPU you just hide. Check your BIOS options. Also check the c…

[privacyuser]

It seems to have hidden the gpu properly now, but it won’t boot with both monitors plugged in. If I plug in the monitor after boot it won’t detect any input. (But it is hidden from dom0, so did it work)? Do I just continue following the guide from here or did it still not work?

[privacyuser]

lspci still shows the Nvidia card and it’s audio device. Is that normal?

[apparatus]

@privacyuser

If I plug in the monitor after boot it won’t detect any input. (But it is hidden from dom0, so did it work)? Do I just continue following the guide from here or did it still not work?

Yes.

@privacyuser

lspci still shows the Nvidia card and it’s audio device. Is that normal?

Yes, the driver should be pciback instead of nouveau or nvidia.
You can check it in the output of:

lspci -k

[privacyuser]

Yes, nvidia driver shows pciback. I created a clone of Debian 11, set the kernel to be provided by quee, HVM, 8 gb ram and 6 cores. Installed nvidia-detect and then used that to install drivers and says drivers are installed. however I don’t notice any performance difference…
Settings manager says DUMMY0
Did I do something wrong?

[apparatus]

Did you follow this guide?

Create a Gaming HVM

Hardware To have an ‘HVM’ for gaming, you must have A dedicated GPU. By dedicated, it means: it is a secondary GPU, not the GPU used to display dom0. In 2023, ‘Nvidia’ and ‘Amd’ GPU work. Not tested with Intel GPUs. External GPU using thunderbolt work (Create a Gaming HVM - #8 by solene) A screen available for the gaming ‘HVM’. (It can be a physical monitor or just to have multiple cables connected to the screen and switching between input source) Dedicated gaming mouse and keyboard. …