Curious about best practices for using qubesOS while also working on some ML projects

User Support
7

Yeah, GPU passthrough guides/posts are spread everywhere :confused:

I mainly used these ones to setup mine:

I took some notes since I prefered patching /usr/share/qubes/templates/libvirt/xen.xml rather than stubdom-linux-rootfs.gz because it seemed simpler.

You can find them below (quickly translated from French to English). I use this technique (and a friend of mine as well) to set up our GPU passthrough and it works fine. Basically you just need to apply a patch to a specific file, set a “qvm-preference” to the qube and then attach the GPU. The rest is to set up Nvidia drivers in the qube, I left it there since you may find it useful.

Do not hesitate if you have problems with the setup.

Infos

Works well but requires a patch in order to use a virtual machine with more than 3GB of RAM.
The Github issue following this problem : https://github.com/QubesOS/qubes-issues/issues/4321

Patch

In order to have GPU passthrough in a virutal machine with more than 3GB of RAM, you have to apply a simple patch in /usr/share/qubes/templates/libvirt/xen.xml (will need to be redone after each update of this file).

In the file /usr/share/qubes/templates/libvirt/xen.xml (remember to backup the file before doing any modification, just in case):

$ diff xen.xml.bck xen.xml
174c174
<                             ,netmask={{ vm.netmask }}"
---
>                             ,netmask={{ vm.netmask }}{% if vm.features.get('gpu-passthrough', False) %} -machine xenfv,max-ram-below-4g=3.5G{% endif %}"
180c180
<                             ,netmask={{ vm.netmask }}"
---
>                             ,netmask={{ vm.netmask }}{% if vm.features.get('gpu-passthrough', False) %} -machine xenfv,max-ram-below-4g=3.5G{% endif %}"

NOTE: the option xenfv is probably not useful.

This patch adds parameters to qemu when the feature (set with qvm-features) gpu-passthrough is set for the qube (it prevents messing with the classical qubes). This is done in the emulator section (that controls qemu)

Then, in dom0, you just need to add the feature to the qube you want to passthrough the GPU:

qvm-features myqube gpu-passthrough 1

Install Nvidia drivers in a standalone qube

Note: it is probably possible to do it in an AppVM by installing the drivers directly in the template and blacklisting nouveau. Blacklist is probably also possible through a kernel cmdline param like modprobe.blacklist=nouveau.

  • Choose a default template
  • Follow the procedure to have the kernel “provided by qube”. Cf. Managing qube kernels | Qubes OS
    TLDR:
    • sudo apt install linux-image-amd64 linux-headers-amd64 grub2 qubes-kernel-vm-support
    • sudo grub-install /dev/xvda
  • Reboot the VM with the kernel “provided by qube” (Settings - Advanced - Kernel: provided by qube)
  • Install Nvidia drivers (from the non-free repositories of debian):
    • Add the non-free repositories if needed in /etc/apt/sources.list: http://deb.debian.org/debian/ bullseye main non-free contrib
    • Install required packages: apt install nvidia-driver nvidia-cuda-toolkit
    • nouveau driver should automatically be hidden by the packages installed in the previous step.
  • Poweroff the VM, and attach the graphic card via PCI passthrough and add it the preference gpu-passthrough like explained in the Patch chapter.
  • Start the VM
  • Enjoy
2 Likes