Step-by-step nvidia GPU passthrough for cuda/vulkan compute applications

[otter2]

@UberFubar

Revert to get a baseline with just Qubes using the GPU’s and the external HDMI working.

See nouveau feature matrix for your card

@UberFubar

Am I correct in that the driver=pciback describes any HW hidden from Dom0. I ask because I saw in previous Thunderbolt had it in its lspci output.

Yes, I think it’s true

@UberFubar

Slightly off topic, do you know anything about the SeaBIOS used in the HVM’s? I am trying to access it ang only found to use the Esc key but there is not enough splash time.

I don’t think you can do anything there, why do you even need this? Everything should be accessible as a configuration from host.

@UberFubar

Seems I dont even need to “hide” my dGPU from Dom0

This can’t be right, maybe you forgot to re-generate grub configuration after editing it?

[UberFubar]

My SeaBIOS issue is I have is for example blackarch does not allow some partition format options if it doesnt recognize a EFI boot setting from the bios. Not a total deal killer but would be more flexible for HVM’s.

Here is my current lspci -k output:
01:00.0 VGA compatible controller: NVIDIA Corporation AD103GLM [RTX 5000 Ada Generation Laptop GPU] (rev a1)
Subsystem: Lenovo Device 2307
Kernel modules: nouveau

This is after I removed the previous GRUB settings and regenerated.

[UberFubar]

My only guess is Lenovo says neither GPU is hard wired and uses a onboard the mainboard display controller to manage between the 2. Possibly it is a NVIDIA ADA5000 or similar GPU difference from the RTX and Quadro versions.

[otter2]

@UberFubar

Here is my current lspci -k output:
01:00.0 VGA compatible controller: NVIDIA Corporation AD103GLM [RTX 5000 Ada Generation Laptop GPU] (rev a1)
Subsystem: Lenovo Device 2307
Kernel modules: nouveau

You are missing “Kernel driver in use” here, “Kernel modules” is simply listing available modules, not necessarily what’s in use

[UberFubar]

I saw that and was not sure what it really meant. All starts too make sense. no driver is being loaded so while not “hidden”, it is not accessed and loaded so no crash when attached to HVM. Makes sense with what I was waiting for my DP cable to start troubleshooting. If I do a xrandr, not option showing the Nvidia,

lxinfo | grep -E “OpenGL vender|OpenGL renderer”
OpenGL renderer string: Mesa Intel(R) Graphics (RPL-S)
and

modprobe nvidia
modprobe: FATAL: Module nvidia not found in directory /lib/modules/6.12.59-1.qubes.fc41.x86_64

and only the laptop display
xrandr | grep “connected”
eDP-1 connected primary 2304x1440+0+0 (normal left inverted right x axis y axis)

[UberFubar]

I thought that possibly with the BIOS set to hybrid simply the Qubes install wasnt loading since I didnt have a external monitor then.

[UberFubar]

I started a new Topic (What I should have done to begine with) here Troubleshoot nouveau on Lenovo P16 Gen 2 w/Nvidia ADA5000 trouble R4.3-RC4

I woud like to clean this up and move my thread to it. Can you point me to anything on explaining on how to do it. Will search also.

[CityDog]

[SOLVED]
Solution: Doing the same for 01:00.1 aka the GPU’s built in audio controller.
So adding it to sys-gpu’s assigned devices along with adding it in grub / hiding it from dom0.

Running Qubes via iGPU works fine for me by simply plugging my monitor to the motherboard HDMI.

But passing the dGPU to sys-gpu crashed my system at sys-gpu boot.

So I edited grub according to the instructions. Now booting does not crash the entire system, but sys-gpu will not boot.

Instead I get this notification after attempting to boot sys-gpu:

Qube Status: sys-gpu
Qube sys-gpu has failed to start: internal error: Unable to reset PCI device 0000:01:00.0: internal error: Active 0000:01:00.1 devices on bus with 0000:01:00.0, not doing bus reset

System info:

CPU: Intel(R) Core™ i7-7700 CPU @ 3.60GHz
Chipset: Intel Corporation Xeon E3-1200 v6/7th Gen Core Processor Host Bridge/DRAM Registers [8086:591f] (rev 05)
Graphics: Intel Corporation HD Graphics 630 [8086:5912] (rev 04) (prog-if 00 [VGA controller])
NVIDIA Corporation GP106 [GeForce GTX 1060 6GB] [10de:1c03] (rev a1) (prog-if 00 [VGA controller])

RAM: 65402 Mb

QubesOS version: R4.3-rc4
BIOS: 1302
Kernel: 6.12.59-1
Xen: 4.19.3

motherboard: asus prime Z270-A

grub:

GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=false
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="rd.luks.uuid=luks-4b701b3e-46cb-4f45-8f8f-6ac95cbf5501 rd.lvm.lv=qubes_dom0/root rd.lvm.lv=qubes_dom0/swap plymouth.ignore-serial-consoles rhgb quiet"
GRUB_DISABLE_RECOVERY="true"
GRUB_THEME="/boot/grub2/themes/qubes/theme.txt"
GRUB_CMDLINE_XEN_DEFAULT="console=none dom0_mem=min:1024M dom0_mem=max:4096M ucode=scan smt=off gnttab_max_frames=2048 gnttab_max_maptrack_frames=4096"
GRUB_DISABLE_OS_PROBER="true"
GRUB_CMDLINE_LINUX="$GRUB_CMDLINE_LINUX rd.qubes.hide_pci=01:00.0 usbcore.authorized_default=0"
. /etc/default/grub.qubes-kernel-vm-support

$ lspci | grep VGA
00:02.0 VGA compatible controller: Intel Corporation HD Graphics 630 (rev 04)
01:00.0 VGA compatible controller: NVIDIA Corporation GP106 [GeForce GTX 1060 6GB] (rev a1)

$ lspci -nn | grep "NVIDIA"
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP106 [GeForce GTX 1060 6GB] [10de:1c03] (rev a1)
01:00.1 Audio device [0403]: NVIDIA Corporation GP106 High Definition Audio Controller [10de:10f1] (rev a1)

Can I have help?

Trying to boot via CLI

$ qvm-start sys-gpu
Error: Start failed: internal error: Unable to reset PCI device 0000:01:00.0: internal error: Active 0000:01:00.1 devices on bus with 0000:01:00.0, not doing bus reset, see /var/log/libvirt/libxl/libxl-driver.log for details

$ sudo cat /var/log/libvirt/libxl/libxl-driver.log
(I removed entries older than today)
2025-12-15 07:43:10.756+0000: libxl: libxl_pci.c:1590:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:00:14.0
2025-12-15 07:50:40.584+0000: libxl: libxl_pci.c:1590:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:00:14.0
2025-12-15 07:50:40.707+0000: libxl: libxl_pci.c:2308:pci_remove_detached: Domain 2:xc_physdev_unmap_pirq irq=16: Invalid argument
2025-12-15 07:51:05.549+0000: libxl: libxl_event.c:855:libxl__ev_xswatch_deregister: remove watch for path @releaseDomain: Bad file descriptor
2025-12-15 07:52:00.250+0000: libxl: libxl_pci.c:1590:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:00:14.0

[Privacc]

I’m getting stuck after installing the Nvidia drivers. I reboot and I can’t launch any programs (not even the command line).

I can see the qube is running in the background from its CPU usage, but nothing is responding. I tried installing both the open source and Nvidia driver with the same result.

I suspect that the dGPU is now handling the display server.

[otter2]

First get a console in your qube and make sure that nvidia driver is actually used. Second, delete nvidia x config: /usr/share/X11/xorg.conf.d/nvidia.conf

[Privacc]

It didn’t make a difference. Maybe I should mention that I’m on Fedora 43.