Goal: passing Gigabyte GeForce RTX 5080 OC GAMING gpu to a Windows 10 qube.
I’ve been following the guides to make a gpu passthrough with initial success. I’ve hidden the GPU from Dom0 and Qubes uses only iGPU right now. I can attach my GPU to the Windows qube and the system “sees” it, can install nVidia drivers for it but it never starts and comes up with a Code: 43 error.
I’ve read in various external guides that newer GPUs now need to have their vbios edited otherwise they will not start in a VM. I did dump the vbios via GPU-Z, used hex edit and placed the edited file in /usr/share/vgabios/GB203.rom in Dom0. Now I’ve added my gpu to the windows-10.xml using this method: Custom libvirt config — core-admin mm_71b5ab21-0-g71b5ab2 documentation.
The output of /etc/qubes/templates/libvirt/xen/by-name/windows-10.xml is
{% block devices %}
{{ super() }}
<hostdev mode='subsystem' type='pci' managed='yes' nostrictreset='yes' permissive='yes'>
<source>
<address domain='0x0000' bus='0x01' slot='0x0' function='0x0'/>
</source>
<rom file='usr/share/vgabios/GB203.rom'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes' nostrictreset='yes' permissive='yes'>
<source>
<address domain='0x0000' bus='0x01' slot='0x0' function='0x1'/>
</source>
<rom file='usr/share/vgabios/GB203.rom'/>
</hostdev>
{% endblock %}
This works i.e. it attaches the pci devices to the qube and the rom entry exists in the windows-10.xml qubes recreates. But I’m not sure if the vbios is read because the results are the same as using qvm-pci a or the qube manager GUI. Will this load the vbios file from this location (Dom0) or does it need to be sent to the qube to be read? Does Qubes even support this libvirt feature? Thanks in advance for any answers.
It is weird that you need to edit you vbios. Multiple causes of Error 43 are described on the arch wiki page. Have you tried their solutions?
As far as I can tell vbios edit is only required on single gpu setups
Yeah, qubes uses libvirt, but it is xen under the hood, expect it to not behave exactly the same as libvirt on KVM.
Thanks for these resources!
The tool they recommend rom-parser gave me the output of:
Valid ROM signature found @36800h, PCIR offset 0h
Invalid PCIR signature: U�
That’s for the original untampered rom. Same results with the roms downloaded from TechPowerUp site.
The modded stripped version I am using shows this:
Valid ROM signature found @0h, PCIR offset ca0h
PCIR: type 0 (x86 PC-AT), vendor: 10de, device: 2c02, class: 030000
PCIR: revision 0, vendor revision: 1
Valid ROM signature found @fa00h, PCIR offset 520h
PCIR: type 3 (EFI), vendor: 0000, device: 0000, class: 000000
PCIR: revision 3, vendor revision: 0
EFI: Signature Valid, Subsystem: Boot, Machine: X64
Last image
Seems ok but these 0000 entries bother me. After loading the rom I seem to have gotten a bit farther inside Windows. When installing nVidia drivers it prompts me to reboot which it did not before and there is no more “device not started” message in the driver event log. The code 43 message is still present. I have disabled resizable BAR in the bios but still no success. One other thing I’ve noticed is that permissive='yes' setting does not get set in the windows-10.xml. Only nostrictreset='yes' is present.
Some more trials in trying to make nVidia drivers start.
I’ve been trying several of the solution from that arch wiki page.
I have added some legacy solutions to the VM’s xml file:
{% extends 'libvirt/xen.xml' %}
{% block features %}
{{ super() }}
<hyperv mode='custom'>
<vendor_id state='on' value='1234567890ab'/>
</hyperv>
<kvm>
<hidden state='on'/>
</kvm>
<ioapic driver='kvm'/>
{% endblock %}
{% block cpu %}
{{ super() }}
<feature policy='disable' name='amd-stibp'/>
{% endblock %}
{% block devices %}
{{ super() }}
<hostdev mode='subsystem' type='pci' managed='yes' nostrictreset='yes' permissive='yes'>
<source writeFiltering='no'>
<address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</source>
<rom file='/usr/share/vgabios/GB203.rom'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes' nostrictreset='yes' permissive='yes'>
<source writeFiltering='no'>
<address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
</source>
</hostdev>
{% endblock %}
I have no idea if they do anything on Qubes tbh.
I’m testing more with the BAR settings right now. I wonder if virtual bar is the culprit for code 43 in this case. From the arch wiki page I’ve tried some methods to manipulate it. Disabling resizable BAR in bios sets both physical and virtual BAR to 256MB. Enabling it sets physical BAR to 16GB.
This is the only method to change physical BAR size that worked:
sudo sh -c 'echo "0000:01:00.0" > /sys/bus/pci/devices/0000:01:00.0/driver/unbind'
sudo sh -c 'echo 14 > /sys/bus/pci/devices/0000:01:00.0/resource1_resize'
sudo sh -c 'echo "0000:01:00.0" > /sys/bus/pci/drivers/pciback/bind'
However only resource1_resize is available. There is no other *_resize. Physical BAR size can be set as low as 64MB but I cannot change the virtual BAR which persists on 256MB - in the wiki page they want it at 8MB. Any ideas welcome…
Only months later I came upon a solution in a comment on the QubesOS github page.
It turns out vBIOS was not the problem at all. The drivers for the GPU kept failing because on the GeGorce 50 series there is some conflict where the vga adapter will fail if you pass it together with the audio part. Yes, despite being in the same IOMMU group and against the usual recommendations on these cards passthrough works only with graphical part used alone. When I detached NVIDIA HD Audio Controller from the VM everything magically started working on both Windows and Linux based qubes.
Another important lesson from that github comment is that to check if the configuration is working you must restart the VM with the attached GPU twice. That is start it, shutdown it, and then start it again. Otherwise you will get false failures on valid setups. That can be a source of endless frustration.
Posting this for everyone who struggles like me with the NVIDIA GeForce 50 series.
TLDR if you have nVidia GeForce 50 series card:
- only pass through the graphic adapter device from your GPU
- restart the VM twice when changing configurations to avoid false failures