I’m sooooooo close but not yet there. Ironically, NVIDIA gpu (RTX3080, 10GB) passthrough works fine in my Windows 10 VM. However, AMD gpu (7900 dgpu) passthrough I’ve not been able to get working. Currently, the passthrough is working, GPU is being used when I launch graphic-intensive programs. However, connecting a monitor to the gpu does nothing, the monitor is not recognized and receives no output. There is output to the monitor during the boot process so I know the monitor and connection are fine. Occasionally, I will get a corrupted-looking color bar-ish picture on the monitor connected to the dgpu. I use uefi boot so I can use the 7000 series igpu for the qubes monitors and save the dgpus for passthrough. I’m mildly paranoid that the source of my troubles is that qubes or xen or efi is doing something to the amd dgpu (even though it is supposedly hidden) when it loads the amdgpu driver since the igpu uses it.
The windows VM shows two displays–the normal window in qubes and a (not viewable) AMD vDisplay, which someone on the internet says it what happens when the AMD gpu is not intialized properly because of the AMD gpu reset bug.
What I’ve done:
- Edit /etc/default/grub to include rd.qubes.hide_pci, iommu=pt, video=vesafb:off video=efifb:off video=simplefb:off nofb initcall_blacklist=sysfb_init (all the frame buffer stuff is a precaution since I noticed without it I would get text on the monitor connected to the gpu indicating memory being initialized.) See Explaining CSM, efifb=off, and Setting the Boot GPU Manually - The Passthrough POST for some background on csm vs uefi and how it affects gpu passthrough.
- Edit /usr/share/qubes/templates/libvirt/devices/pci.xml to pass a techpower rom file (didn’t seem to help). For those who care, I filtered on vm.name and device.libvirt_name=‘pci_0000_05_00_0’.
- Edit /usr/share/qubes/templates/libvirt/devices/pci.xml to set rom bar=off (didn’t see to help)
- Edit /usr/share/qubes/templates/libvirt/xen.xml to disable the cpu hypervisor feature (no noticeable affect)
- Create an sh file to change bar0 to 32GB and bar2 to 8MB. This is apparently necessary for Windows to avoid error 43 for whatever reason. Enabling or disabling rebar in bios does not seem to change behavior (may change performance) [SOLVED] 7900 XTX Code 43 or How to get 7900 XTX to work on VFIO - Virtualization - Level1Techs Forums
- Create an sh file to remount the gpu (remove and rescan).
- Attach the GPU with permissive and no-strict-reset. Didn’t seem to change behavior.
- Enable everything related to virtualization in my motherboard bios (proart x670e-creator)
- Disable motherboard and cpu ASPM in my motherboard bios. Didn’t seem to change behavior.
Supposedly qubes 4.3 will contain fixes(?) for amdgpu passthrough. However, seabios is broken in the current testing repos (probably [Build 202412140459-4.3] start_standalone_with_cdrom_vm test fails on system installed in legacy boot mode · Issue #9645 · QubesOS/qubes-issues · GitHub) so I’m not able to test.
Anyway, thought I’d throw this out there since I could use some help.
Future plans unless something changes:
- try modprobe blacklisting (https://www.reddit.com/r/Proxmox/comments/15kbmyh/working_rx_7900_xt_single_gpu_passthrough_to/)
- try booting into windows natively and using capturing the vbios from it instead of using the one from techpower. I think I remember reading somewhere that the vbios is different (i.e. modified on initialization) depending on whether the gpu is loaded natively from linux or windows and that it needs to match the OS you are passing it through to. (perhaps from VFIO Single GPU Passthrough Guide on Linux | n0k0m3 Dev Blog)
- look more into the amd gpu reset bug (Working around the AMD GPU Reset bug on Proxmox using vendor-reset – Nicholas Sherlock)
- look at memory usage more closely (
cat /proc/iomem, proxmox7-win11-gpu-passthrough.md · GitHub)