What graphics devices do you have?
Nvda rtx 3060 and intel i5 11th gen integrated graphics
Oh cool I have an RTX 3060 as well, but AMD’s integrated graphics. Before blacklisting the dgpu, could qubes detect both graphics devices?
How do i check?
I went into qube manager settings for one of my templates. In the devices tab i can see a bunch of tiger lake devices. Unsure which is integrated graphics tho. I do see my rtx 3060 and nvidia audio driver
You run ‘lspci’ in dom0 and look for graphics adapters (they’re called VGA adapters I think). Is your igpu connected to an output?
I only see 1 VGA and its nvidia. What do you mean by igpu connected to an output?
As per the beginning of the original post, both your dgpu (nvidia) and igpu (intel) need to be connected to something. If your monitor has two inputs, then you can connect both your igpu and dgpu to your monitor (now depending on your motherboard you might have to do some configuration. On my system I need to go into the bios and set the igpu as my primary video adapter otherwise the system won’t POST).
Also be prepared to reset your motherboard if anything goes wrong. Research that beforehand.
No but seriously I’m not doing gpu passthrough anymore because after setting it up and using it a few times, one morning my monitor went blank while trying to boot up the qube that was supposed to have the gpu passed through to it, so I shut my PC down using the power button. On booting it back up (and changing to the DGPU output) I got this: https://youtu.be/8yfwn20XUes Don’'t know what caused it specifically (might be a bad HDMI cable, who knows) but yeah be careful. Fortunately my motherboard reset itself after failing to POST a few times.
By connected to something do you mean switching between switchable graphics and discrete graphis in UEFI settings?
for resetting my motherboard, does BIOS self healing or BIOS back flash in uefi settings do anything?
I switched to switchabe graphics in uefi and now i see 2 vga. One intel and one nvidia.
Edit: second monitor works
Okay so this completely went over my head. My laptop has a mux switch and i was using discrete graphics this whole time. does this mean i was using my nvidia card instead of integrated intel gpu?
I got my gpu passthrough to work despite skipping the ‘integration with qubes os’ part. Is that necessary?
How do i increase ram on my standalone? Its greyed out as 4000mb in qubes settings.
Gpu passthrough only works when switchable graphics are selected instead of discrete graphics on uefi settings (to show both gpus obviously(
However external monitor doesnt work unless…
Either
-
Discrete graphics is set
-
Or switchable graphics is set but rd.qubes.hide-pci:0a:00.0 is removed from kernel command line at boot
is it not possible to use gpu passthrough and external device at the same time?
im stuck between choosing the above two because i want to use 2nd monitor
Edit: just tested suspend to RAM and strangely it works! Never worked for me before.
Did you shutdown the qube before trying to change the memory?
You want to use external monitor in dom0?
Maybe the external monitor port is wired directly to dGPU so if you passthrough dGPU to the qube then you won’t be able to use external monitor in dom0.
But you can try to passthrough iGPU to the qube instead of dGPU and use dGPU in dom0, then you may be able to use external monitor in dom0.
You can use virtualgl in your qube with GPU passthrough so you won’t need separate monitor for it:
But it’s all is just a guess.
Thanks. I will have to investigate further.
What is the purpose of this step exactly?
Yes i did. Its powered ofd but still greyed out
You need to tell Xorg how it should use your GPU and monitor.
Are you trying to change the “Max memory” and not “Initial memory”?
If you disable memory balancing (unchecked “Included in memory balancing”) then “Max memory” will be grayed out because it’s not used.
It’s better to disable the memory balancing for HVM qubes with attached PCI devices since it could cause some problems with them.
Hi,
I’ve sort of successfully passed through my GPU, everything works post boot, however the problem is getting past the boot screen.
I’ve found I can only login if I disconnect the monitor from my passed-through GPU, then reconnect after login.
I’ve traced this to Xorg erroring, and stopping LightDM from running.
Xorg Log (pwd= qubes):
It looks like Xorg is trying to configure settings with Nvidia / Nouveau drivers, when I have passed through my Nvidia GPU.
My current setup is that I have one monitor plugged straight into the motherboard using my AMD CPUs integrated graphics, and an external monitor plugged into my passed-through Nvidia GPU dedicated to the Gaming Qube.
I thought the solution would be to simply disable Nouveau drivers from loading, so have applied the following parameters to the Kernel Commandline:
GRUB_CMDLINE_LINUX="… rhgb quiet rd.qubes.hide_pci=01:00.0 nouveau.modeset=0 rd.driver.blacklist=nouveau "
Even after these changes, it seems Xorg is still trying to work with the nouveau drivers, and giving me the same error on login.
I’ve seen there is mention of editing/creating Xorg.conf files, however I haven’t done so as it looked like that was meant for the Gaming Qube, rather than dom0.
Any help would be appreciated thank you.
I was able to get things to work without any apparent issues using a nvidia GPU, but I’m running into issues with an AMD GPU. Please let me know if you have any ideas, suggestions, or if there’s any other info that I can provide which might be helpful.
The GPU is an rx 6750 xt and in its own IOMMU group, it’s using the same slot that the nvidia GPU was using. From dom0, rd.qubes.hide_pci is used to hide both the GPU and its audio controller, which both report pciback in lspci -k. Both are being passed to the guest. I’ve tried several different guests and run into various issues.
I am able to boot bare metal to a live Fedora ISO with gnome. It doesn’t appear to have any issues, but I haven’t done much with it.
fedora-xfce template
The VM terminal with qubes integration crashes soon after opening if the GPU is passed through. I wasn’t able to fix this. I tried to install drivers from AMD’s website, but the RHEL drivers didn’t work and there aren’t any Fedora specific drivers that I was able to find from AMD.
debian-xfce template
Just like with Fedora, the VM terminal with qubes integration crashes soon after opening if the GPU is passed through. I was able to stop this from happening by installing drivers from AMD’s website. Following this guide, I am able to get the qube to output to the display attached to the GPU. I can reliably start i3, but not xfce.
I have gotten xfce to start a few time, but it has been extremely slow. Sometime xfce seems to just get stuck as a cursor on a black screen. When it has started, it loads in very slowly and is very laggy. Mouse movement is fine, but any sort of interactions like opening menus or clicking and dragging are very very slow. This leads me to believe that the times xfce gets “stuck” it’s actually just taking a long time to load. I haven’t tried using i3, but I assume that it is impacted by the same performance issues and is just able to load because it’s more lightweight or something.
I’ve tried running top and nvtop in the qubes integrated display while trying to run xfce on the GPU and neither shows significant resource usage. CPU usage is fairly low and nvtop shows GPU usage at 0%.
This seems like a driver issue, but I’m not sure how to verify this or what to do. lspci -k shows that the amdgpu driver is in use for the GPU.
archlinux community template
I’m not familiar with arch. The template has issues updating that I’m not sure how to correctly fix, but I was able to get it to update by disabling signature verification. Unfortunately I wasn’t able to figure out how to install a kernel in arch and one doesn’t seem to be included so I wasn’t able to boot with kernel provided by the qube which stopped me from getting very far.
Pop!_OS 22.04 LTS
Using the “normal” (not nvidia, not ras pi) image.
Live
Booting off the ISO with the qube’s display disabled never finishes loading the display. There is a bunch of error spam on the display of the same few errors:
amdgpu 0000:00:05.0: [drm] *ERROR* [PLANE:70:plane-5] commit wait timed out
amdgpu 0000:00:05.0: [drm] *ERROR* [CRTC:91:crtc-0] flip_done timed out
amdgpu 0000:00:05.0: [drm] *ERROR* flip_done timed out
amdgpu 0000:00:05.0: [drm] *ERROR* [CRTC:91:crtc-0] commit wait timed out
amdgpu 0000:00:05.0: [drm] *ERROR* flip_done timed out
Eventually it gives up and at this point I’m able to login through the console provided within qubes. The display output never gets to a console or GUI.
Installed
First I install Pop!_OS without passing through a GPU. Then I edit the grub commandline. I’ve tried both pci=nomsi and pci=nomsi amdgpu.dcdebugmask=0x10. Then I remove the qubes display using qvm-features qube video-model none and boot with the GPU attached. I get a gray screen with a cursor, but that’s it. Eventually I’m able to login with the console within qubes and it seems to be having the same issue as the live distribution. Even when I’m able to make a console connection, the display is unchanged.
manjaro-xfce-24.0.7
I tried this booting off the ISO with the qube’s display disabled. There was output to the physical display through the passedthrough GPU, but it wasn’t able to load the GUI. I don’t remember what the exact issue was but I think it had the same flip_done timeout errors that I described earlier.
windows 11
My primary goal is to get any linux distribution working with passthrough, but I’m including this here for completeness and in case it somehow helps identify the issue.
I don’t remember exactly how I tested windows 11 (qubes display on or off) but I wasn’t ever able to get a display output when booting from the install media. Attaching the GPU to a qube with windows already installed showed no GPU device in device manager, which seems strange. The AMD driver installer refused to install, saying something along the lines of it not being an AMD system.
I have a laptop with dedicated 3080 GPU (Legion 7i 16), I use a second external monitor but the display ports on my laptop are hardwired to my GPU. So when I hide my GPU in Dom0, I can not use my secondary monitor. Is there a workaround for this issue?
It look like an mistake from the motherboard and the bios.
Check in your bios if you can configure what is the primary GPU (some bios have this option. Some don’t.)
If your motherboard doesn’t have this explicit option, you can try to switch HDMI output to DisplayPort / or something else: some bios have an implicit primary GPU configuration based on the output type.
If none of that work, only workaround that I see is to use an HDMI splitter / DisplayPort splitter, and make sure that at boot, the splitter send the data to nothing, so the motherboard will detect that nothing is connected to your DGPU and implicitly set the primary GPU to the other GPU
1 Like
I’ve already set the default graphics output on my BIOS to be the Integrated Graphics, however looking it up it seems there may be a bug with the BIOS Firmware.
I’m now using a HDMI splitter and it works perfectly.
I have Input 1 on the splitter connected to my IGPU, and Input 2 connected to my passed-through GPU, so whenever I boot up the Gaming HVM I just need to press a button on the splitter.
Thank you very much for suggesting the splitter solution, completely forgot I had one.
Following my talk at QubesOS summit 2024, added my PoC project to try to automate that:
https://git.sr.ht/~yukikoo/gpu_template
And the slides I used for the presentation are available here: https://neowutran.ovh/QubesOS_2024_presentation.zip
4 Likes