I was able to get things to work without any apparent issues using a nvidia GPU, but I’m running into issues with an AMD GPU. Please let me know if you have any ideas, suggestions, or if there’s any other info that I can provide which might be helpful.
The GPU is an rx 6750 xt and in its own IOMMU group, it’s using the same slot that the nvidia GPU was using. From dom0, rd.qubes.hide_pci is used to hide both the GPU and its audio controller, which both report pciback in lspci -k. Both are being passed to the guest. I’ve tried several different guests and run into various issues.
I am able to boot bare metal to a live Fedora ISO with gnome. It doesn’t appear to have any issues, but I haven’t done much with it.
fedora-xfce template
The VM terminal with qubes integration crashes soon after opening if the GPU is passed through. I wasn’t able to fix this. I tried to install drivers from AMD’s website, but the RHEL drivers didn’t work and there aren’t any Fedora specific drivers that I was able to find from AMD.
debian-xfce template
Just like with Fedora, the VM terminal with qubes integration crashes soon after opening if the GPU is passed through. I was able to stop this from happening by installing drivers from AMD’s website. Following this guide, I am able to get the qube to output to the display attached to the GPU. I can reliably start i3, but not xfce.
I have gotten xfce to start a few time, but it has been extremely slow. Sometime xfce seems to just get stuck as a cursor on a black screen. When it has started, it loads in very slowly and is very laggy. Mouse movement is fine, but any sort of interactions like opening menus or clicking and dragging are very very slow. This leads me to believe that the times xfce gets “stuck” it’s actually just taking a long time to load. I haven’t tried using i3, but I assume that it is impacted by the same performance issues and is just able to load because it’s more lightweight or something.
I’ve tried running top and nvtop in the qubes integrated display while trying to run xfce on the GPU and neither shows significant resource usage. CPU usage is fairly low and nvtop shows GPU usage at 0%.
This seems like a driver issue, but I’m not sure how to verify this or what to do. lspci -k shows that the amdgpu driver is in use for the GPU.
archlinux community template
I’m not familiar with arch. The template has issues updating that I’m not sure how to correctly fix, but I was able to get it to update by disabling signature verification. Unfortunately I wasn’t able to figure out how to install a kernel in arch and one doesn’t seem to be included so I wasn’t able to boot with kernel provided by the qube which stopped me from getting very far.
Pop!_OS 22.04 LTS
Using the “normal” (not nvidia, not ras pi) image.
Live
Booting off the ISO with the qube’s display disabled never finishes loading the display. There is a bunch of error spam on the display of the same few errors:
amdgpu 0000:00:05.0: [drm] *ERROR* [PLANE:70:plane-5] commit wait timed out
amdgpu 0000:00:05.0: [drm] *ERROR* [CRTC:91:crtc-0] flip_done timed out
amdgpu 0000:00:05.0: [drm] *ERROR* flip_done timed out
amdgpu 0000:00:05.0: [drm] *ERROR* [CRTC:91:crtc-0] commit wait timed out
amdgpu 0000:00:05.0: [drm] *ERROR* flip_done timed out
Eventually it gives up and at this point I’m able to login through the console provided within qubes. The display output never gets to a console or GUI.
Installed
First I install Pop!_OS without passing through a GPU. Then I edit the grub commandline. I’ve tried both pci=nomsi and pci=nomsi amdgpu.dcdebugmask=0x10. Then I remove the qubes display using qvm-features qube video-model none and boot with the GPU attached. I get a gray screen with a cursor, but that’s it. Eventually I’m able to login with the console within qubes and it seems to be having the same issue as the live distribution. Even when I’m able to make a console connection, the display is unchanged.
manjaro-xfce-24.0.7
I tried this booting off the ISO with the qube’s display disabled. There was output to the physical display through the passedthrough GPU, but it wasn’t able to load the GUI. I don’t remember what the exact issue was but I think it had the same flip_done timeout errors that I described earlier.
windows 11
My primary goal is to get any linux distribution working with passthrough, but I’m including this here for completeness and in case it somehow helps identify the issue.
I don’t remember exactly how I tested windows 11 (qubes display on or off) but I wasn’t ever able to get a display output when booting from the install media. Attaching the GPU to a qube with windows already installed showed no GPU device in device manager, which seems strange. The AMD driver installer refused to install, saying something along the lines of it not being an AMD system.