Another 2. GPU passthrough post

Rnd3sB3g13rng · February 14, 2021, 11:34pm

Hello all,

my big dream to work completely on Qubes is within reach.
For the work I need programs that need to be accelerated on the hardware side.

Amd ryzen 7 5800x
rog strix x570-F gaming
amd radeon rx 580 (1. gpu)
amd radeon rx vega 64 (2. gpu)
2*32 GB ddr4 ram
1tb m2 ssd
If necessary I can provide more system specs.

Since the attempt to install qubes 4.0.3 failed because of a blackscreen at boot time I use qubes 4.1 (Signed Q4.1 alpha iso).

I would like to detach the 2nd GPU from dom0 and assign to a hvm.
I have read various guides and failed with all of them.
In detail I wonder what went wrong with https://groups.google.com/g/qubes-users/c/zHmaZ3dbus8/m/rWll-ywQCAAJ:

I first got the device id for my second GPU:

[user@dom0 ~]$ qvm-pci
...
dom0:0c_00.0   VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Vega 10 XL/XT [Radeon RX Vega 56/64]                 
dom0:0c_00.1   Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Vega 10 HDMI Audio [Radeon Vega 56/64]
...

Then edit grub: (I tried both parameters xen-pciback.hide=(0c:00.0)(0c:00.1) and rd.qubes.hide_pci=0c:00.0,0c:00.1) individually and together.)

[user@dom0 ~]$ sudo vim /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="gfxterm"
GRUB_CMDLINE_LINUX="rd.luks.uuid=luks-1b2813e0-e5fd-4dfb-84ad-400aa8f4504f rd.lvm.lv=qubes_dom0/root rd.lvm.lv=qubes_dom0/swap plymouth.ignore-serial-consoles rd.driver.pre=btrfs rhgb quiet xen-pciback.hide=(0c:00.0)(0c:00.1) rd.qubes.hide_pci=0c:00.0,0c:00.1 modprobe=xen-pciback.passthrough=1 xen-pciback.permissive"
GRUB_DISABLE_RECOVERY="true"
GRUB_THEME="/boot/grub2/themes/qubes/theme.txt"
GRUB_CMDLINE_XEN_DEFAULT="console=none dom0_mem=min:1024M dom0_mem=max:4096M ucode=scan smt=off gnttab_max_frames=2048 gnttab_max_maptrack_frames=4096"
GRUB_DISABLE_OS_PROBER="true"
[user@dom0 ~]$ sudo grub2-mkconfig -o /boot/grub2/grub.cfg
[user@dom0 ~]$ reboot

Since the guide uses an Intel processor, I can’t check this 1:1, but here is the output.
[user@dom0 ~]$ xl dmesg
xl-dmesg-output.log (9.0 KB)

So far so good, but then somehow only the vga device is assignable and not the sound device.
[user@dom0 ~]$ xl pci-assignable-list
0000:0c:00.0

I just went ahead and assigned the GPU (both 0c:00.0,0c:00.1 only taking 0c:00.0 results in a driver reset conflict) to the hvm via the qubes manager gui.
Nothing happens at startup, only that the computer slowly freezes. The qrexec seem to be broken. Here are the logs:

guest-gpu-pt-test.log (931.0 KB) guest-gpu-pt-test-dm.log (79.6 KB) qrexec-gpu-pt-test.log (885 Bytes)

I would really appreciate help, but I also know that you have a lot to do. That’s why I would be willing to donate $100 to Qubes if a solution works

Edit: According to some guides I need to edit the hvm kernel, but I have no idea where to start…

Rnd3sB3g13rng · February 15, 2021, 5:34pm

Update:

I’m quite embarrassed to admit it, but after another night of debugging I realized that GRUB_CMDLINE_LINUX was not updated properly…

You can check the parameters with cat /proc/cmdline.
It was not
sudo grub2-mkconfig -o /boot/grub2/grub.cfg
but
sudo grub2-mkconfig -o /boot/efi/EFI/qubes/grub.cfg
necessary. According to (R4.1: Restore Grub2 in UEFI mode · Issue #4902 · QubesOS/qubes-issues · GitHub) updating both is fine.

Now this is also displayed correctly:
[user@dom0 ~]$ xl pci-assignable-list
0000:0c:00.0
0000:0c:00.1

For memory less than 3 gb it works fine. Howerver assigning more memory results in
(2nd GPU passtrough, VM dies on boot. · Issue #5603 · QubesOS/qubes-issues · GitHub).

Rechecking the parameters reveals a passthrough error:
[root@dom0 qubes]# cat /sys/module/xen_pciback/parameters/hide
(0c:00.0)(0c:00.1)
[root@dom0 qubes]# cat /sys/module/xen_pciback/parameters/passthrough
N
[root@dom0 qubes]# cat /sys/modules/xen_pciback/parameters/permissive
Y

Changing modprobe=xen-pciback.passthrough=1 to xen-pciback.passthrough=1
leads to
[root@dom0 qubes]# cat /sys/module/xen_pciback/parameters/passthrough
Y

Now there is still stubdom to change as in (AppVM with GPU pass-through crashes when more than 3.5 GB (3584MB) of RAM is assigned to it · Issue #4321 · QubesOS/qubes-issues · GitHub).

But the location has moved in 4.1:
[user@dom0 stubroot]$ cp /usr/lib/xen/boot/stubdom-linux-rootfs stubroot/stubdom-linux-rootfs.gz
cp: cannot stat '/usr/lib/xen/boot/stubdom-linux-rootfs': No such file or directory

find / -name "*stubdom*"
… there are multiple solutions …
/usr/libexec/xen/boot/qemu-stubdom-linux-rootfs
/usr/lib46/xen/boot/stubdom-linux-rootfs

I took the first one and noticed that the second one was updated aswell.

[user@dom0 ~]$ cp /usr/libexec/xen/boot/qemu-stubdom-linux-rootfs stubroot/stubdom-linux-rootfs.gz
Also you have to change
SP=$'\x1b'
to
SP=$'\n'
in qubes 4.1.
finally change the last command of this guide to
sudo mv ../stubdom-linux-rootfs /usr/libexec/xen/boot/

But now I still got errors. On bootup I got
“Cannot connect to qrexec agent for 60 seconds, see /var/log/xen/console/guest-gpu-pt-test.log for details”

Here are all new logs.

guest-gpu-pt-test.log (1013.3 KB) guest-gpu-pt-test-dm.log (801.2 KB) guid.gpu-pt-test.log (98 Bytes) qrexec.gpu-pt-test.log (143 Bytes)

I would really like to get this working and appreciate any help:)

Update 2:
Installing amd drivers according to (Qubes OS article) and using a hvm with no kernel, results in a different error:
Instantly getting (and not after a minute or so) I got the message
“qrexec-daemon startup failed: 2021-02-15 18:59:50.259 qrexec-daemon[26084]: qrexec-daemon.c:134:sigchld_parent_handler: Connection to the VM failed”.

rasta · March 24, 2021, 8:24am

I cannot get Gpu passthrough working, too. Neiter in dom0 and in a fedora vm for example. Trying to attach it with the qubes manager then my laptop goes black and do a restart. Here is a screenshot of my pci devices. Can anybody explain me the right devices I need to attach to the vm? But first it must be work in dom0 or? Otherwise it is not possible in a vm? Sorry I don*t understand that really how to do it, but I try to learn;)
Maybe fepitre has soon the time to tell us how sys-gui-gpu is working for him with the nvidia 960 gtx card. That should be able to work at our systems as well and then finally we’ll get gpu passthrough working. The topic “gpu” is sooo complex

neowutran · March 25, 2021, 8:04pm

Hi,
Do you still have this issue ?
Out of curiosity, why “Also you have to change
SP='\x1b' to SP=’\n’” ?
What are your Qube settings (ram amount / pci passthou strict reset / pci passthrou devices / … ) ?
I am not yet using R4.1 on my desktop, so haven’t tested how it work on R4.1 yet

ro0t · March 28, 2021, 11:50am

I managed yesterday to GPU passtrough my Nvidia GTX 1650 Max-Q to my Manjaro HVM.

I see my GPU in Manjaro Hardware Settings and I can install nVidia Drivers.

When I restart my VM, it booted until Desktop and picture goes black, but VM does not crashes like every single time before.

I looked in dmesg and I noticed one error at the end:

nvidia: module verification failed: signature and/or required key missing - tainted kernel

Now my question is:

How to disable any possibility for a VM to know that it is a VM ?

I want that my Manjaro VM sees:

A real CPU, not Xen-cpu
A real GPU, not virtualised Hardware piece

In dmesg I also saw that my Manjaro VM sees my GPU on slot 00:07.0
My GPU is in slot 01:00.0

I thing now my problem is that my VM is detecting itself, and nvidia is not loading because of it.

edit:

dmesg | grep nvidia

nvidia: loading out-of-tree module taints kernel.
nvidia: module licence nvidia taints kernel.
nvidia: module verification failed: signature and/or required key missing - tainted kernel
nvidia-nvlink: nvlink core is being initilized, major device number 238
nvidia 0000:00:07.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=io+mem
nvidia-modset: Loading nvidia kernel mods setting deriver for unix platforms 460.56 tue fed 23 23:230…
[drm] [nvidia-drm] [GPU ID 0x00000007] Loading driver
[drm] initillized nvidia-drm 0.0.0 20160202 for 0000:00:07.0 on minor 0

khmartinsen · April 29, 2021, 10:08pm

Are you using the latest Nvidia driver? I was able to passthrough my 1070 without issue besides when the higher ram is assigned. As of right now there is not a way to hide that the GPU is being passthroughed. However that should be an issue with the latest nvidia drivers now.

unjust · May 10, 2021, 1:47pm

I’ve had no luck with patching stubdom in R4.1. The files which resemble the file in the guides are:

/usr/lib64/xen/boot/qemu-stubdom-linux-rootfs
/usr/libexec/xen/boot/qemu-stubdom-linux-rootfs

Performing instructions from the Github issue, also listed on neowutran.ovh, makes HVM’s fail to start after 60 secs.
Logs don’t give any hints here.

AMD Vega is still being passthroughed successfully, although I am running standard Qubes kernel, not VM-installed distro kernel, and windows open with the window manager on my other GPU-connected screen like other PVH VM’s.

I guess then I need to use the VM kernel and make it boot with Grub? Then connect it via the GPU to another monitor.

neowutran · June 21, 2021, 8:46pm

I have a successfull GPU passthrough on R4.1.
I followed my doc but needed to modify the stubdom-linux-rootfs patch.

github.com/QubesOS/qubes-issues

AppVM with GPU pass-through crashes when more than 3.5 GB (3584MB) of RAM is assigned to it

opened 06:48PM - 19 Sep 18 UTC

pqyptixa

C: Xen C: core P: default T: enhancement

### Qubes OS version: R4.0 ### Affected component(s): Debian 10 AppVM -…-- ### Steps to reproduce the behavior: So, I finally could pass a GPU to a VM with PCI passthrough some weeks ago, but for some reason, I can't get it to run with more RAM. I asked #qubes_os @ freenode about this issue, and ended up making a guide on how to get GPU-passthrough done. I'm not going to post the complete guide here, so here's a link to it: https://paste.debian.net/1043341/ . I don't mind if someone else posts it here or elsewhere, but hopefully people will discuss it and improve it. My system: Qubes 4.0 running on a AMD Ryzen 1600 CPU + MSI B350 Mortar motherboard (BIOS 1E0) + ASUS RX560 2GB GPU for the VMs + an old NVIDIA 610 GPU for dom0 (as secondary GPU). ### Expected behavior: AppVM runs just fine with > 3.5GB of RAM. ### Actual behavior: AppVM crashes with more than 3.5GB of RAM. The way it crashes depends on how much memory is assigned to it, but I usually see almost-instant kernel panics, systemd-init crashing, fsck failures and/or other random processes crashing, too. ### General notes: Other than removing "nomodeset" from the kernel command line, everything is set as default. PS: link to the guide expired. Here a a couple of similar guides: https://github.com/Qubes-Community/Contents/blob/master/docs/customization/windows-gaming-hvm.md and https://neowutran.ovh/qubeos.html

Rnd3sB3g13rng · June 27, 2021, 10:04pm

Cool!
Did you overwrite this file?
/usr/libexec/xen/boot/qemu-stubdom-linux-rootfs

neowutran · June 28, 2021, 7:09am

Yes.
Tried to document the steps here: Qubes OS article and here Update Windows Gaming HVM documentation by neowutran · Pull Request #139 · Qubes-Community/Contents · GitHub

unjust · July 16, 2021, 11:36pm

Hi, I read your articles and managed to finally get everything working. Except I cannot get into a 3D world.
I can boot up games and get to the menus, but when trying to join a match or similar, it just gives an infinite loading screen.
Could it have something to do with IOMMU groups? I have almost the same setup as @Rnd3sB3g13rng, with Vega and RXX80.
I couldn’t find any IOMMU groups with my Fedora live USB with the kernel parameters. Maybe with another Linux distro?
I’m not sure if they are in the same IOMMU group because of this.

Could this be the problem here? Everything else works fine, and I can play 2D games using OpenGL with it.

CS:GO throws an error relating to DirectX. With Windows 10 Ameliorated DirectX12 is installed by default.

unjust · July 17, 2021, 2:38am

Solved… by installing Visual C++ redistributables.
Somehow they don’t provide those when you install most games these days.

But there is another issue I have. When “Safely ejecting the Vega GPU” it gives an error instantly after restarting the Windows HVM.
I have to reboot my Qubes after this. I just restart normally now, but after a few restarts the GPU suddenly goes into 100% fan speed, and I have to reboot Qubes. Not a giant issue, though.

unjust · July 26, 2021, 9:40am

There are some very noticeable lags when gaming with my Windows 10 HVM. I guess this is due to me just using a single LVM thin pool, which limits all my drives to the slowest one? These are all SATA/PCIe SSDs.
I haven’t heard anything about PCIe SSDs being mandatory for QEMU/KVM in other distros, but perhaps for Qubes? On your guide you mention M.2 disks, by which you of course mean a M.2 PCIe disk.

neowutran · July 26, 2021, 1:23pm

I can just tell that on my side, I only have 1 pool, containing 2 extremely fast M.2 PCIe disk. Performance are near native, I don’t have any visible lag that I didn’t had before using Qubes. Maybe try to check the gpu driver you are using, but yes, from my test, windows hvm require a very fast disk to not have lags. linux hvm are less an issue.

unjust · July 26, 2021, 5:10pm

How do you use USB passthrough? After a lot of troubleshooting I’ve pinpointed the RAM limit patch to not work with stubdom-qrexec feature. From package xen-hvm-stubdom-linux-full. It’s detailed in this post.
Just Qubes Windows Tools won’t let me USB passthrough.

Edit: Thanks to @jevank I learned I need to patch qemu-stubdom-linux-full-rootfs after installing xen-hvm-stubdom-linux-full. Working now.

unjust · August 1, 2021, 12:12am

Yes, I had a working setup but I ran into a new problem now when I switched my “secondary GPU”, or gaming GPU to my x16 slot, and my dom0 GPU to my x16 (x8) slot, but now I’m having difficulty hiding the gaming GPU from dom0.
It worked then the gaming GPU was in the second x16 (x8) slot.

It will boot with the gaming GPU monitor off when at the disk decryption screen, so it looks like it’s working, but after decrypting, it now turns on the input from the gaming GPU card.

GRUB reads rd.qubes.hide_pci=03:00.0,03:00.1

And a side note, if you have 2 M.2 disks both running in an Ultra slot that’s a damn good build.
And SMT didn’t really provide a benefit in my testing so far.

unjust · August 1, 2021, 11:59pm

I followed your advice and got a hold of a PCIe disk with 2,6GBps read, 1,8GBps write.
Performance is still choppy while gaming for me, and there is intermittent sound lag.

If I stand still in a game, I can reach comfortably 130-40 FPS.
If i start shaking the mouse around it tanks as low as 20-30 FPS.

Using 3 vcpus seems to yield the best, although subpar, performance.

OC’d the GPU a lot, didn’t help much. I think the issue is more related to disk throughput.
And perhaps also worth mentioning is that my GPU is running in x4 mode.
Had problems hiding the x16(x8) slot from dom0.

neowutran · August 10, 2021, 9:30am

Can you run some benchmark ? I am a bit curious to understand the difference of performance between my setup and your setup. To compare some benchmark score to try to understand what is the difference.
And how do you use your mouse ? PCI-passthough of one usb controller ?

neowutran · August 11, 2021, 6:26pm

Depending on your timezone, you could reach me on matrix “neowutran”, “neowutran.ems.host” domain.

equbes · September 20, 2021, 1:18am

I’ve followed the guides (thanks @neowutran for the detailed steps), but sadly the results were bad. First here’s my setup:

AMD Ryzen 9 3900X
ROG Strix x570-E Gaming
64GB ram
2tb m2 nvme ssd

dom0 gpu: NVIDIA 1060
passthrough gpu: NVIDIA 1080 ti

passthrough works in the sense that I could add it to a windows vm and get it running without external issues, but the VM is barely usable at all, with a lot of VIDEO_TDR_FAILURE BSoD errors.

worth mentioning: I was able to use usb passthrough without issues, and I got audio working from the gpu audio device on the monitor speakers