My experience with gpu passthrough (a guide, of sorts)

General Discussion
1

Hardware used:
Ryzen 5 3400G
ASRock B450M Steel Legend
NVIDIA RTX 3060
A monitor with two HDMI inputs; both my motherboard and gpu are connected to the monitor, the primary video adapter is set to the iGPU in the UEFI/BIOS.

I followed this guide: Qubes OS article up to the point where you have to make a new qube. That’s where things got complicated for me.

  1. Fedora template: Have a look at this: NVIDIA GPU passthrough into Linux HVMs for CUDA applications - Community Guides - Qubes OS Forum
    The guide is a bit misleading; you’ll have to use a separate kernel for the qube otherwise the module won’t compile. However even after getting it to use a separate kernel (take a look at Managing qube kernels | Qubes OS on how to do that) it could begin compiling, but it would exit with an error. I gave up on using a fedora template at that point.

  2. Standalone arch iso: Perhaps this is on me but I just downloaded an archlinux iso and booted a qube off of that, except I couldn’t get beyond the grub sreen. Don’t know what I was supposed to do to get it working, but I didn’t pursue it further.

  3. Archlinux community template: To install a custom kernel here I had to take a look at this github issue: archlinux: kernel inside VM not working · Issue #5584 · QubesOS/qubes-issues · GitHub (when installing the initramfs it complained the image wasn’t complete). Anyway after successfully managing to install a kernel I tried both nvidia and nvidia-open, and with both the vm does boot but running nvidia-smi reports ‘No devices were found’. Don’t know what to do there but would appreciate some pointers.

  4. Debian template: This is the one that actually worked for me (not initially though). I managed to install a kernel using the wiki page I linked above, then installed the nvidia drivers related to debain’s wiki page. After that it wouldn’t boot due to some qrexec timeout error. I enabled debug mode in the settings, and that time it did boot. After that it didn’t boot again. Anyway, I increased the initial memory size to 2000 MB and that seems to have fixed it. I can run nvidia-smi, and successfully use nvdec with mpv resulting in half the cpu usage when playing a video. I haven’t tested games/CUDA stuff yet.

But yeah debian is really complicated (to me) and I’d like to know how to get this to work with arch.

3 Likes
2

I am fairly certain that is because default fedora template has only 1 GB dedicated to /tmp/ and that is not enough to build the module. Run mount -o remount,size=5G /tmp/ or something similar to temporarily (until reboot) extend it.

EDIT: Upon further testing I realize that older version of akmods actually fails to build for some reason

1 Like