Goal: passing Gigabyte GeForce RTX 5080 OC GAMING gpu to a Windows 10 qube.
I’ve been following the guides to make a gpu passthrough with initial success. I’ve hidden the GPU from Dom0 and Qubes uses only iGPU right now. I can attach my GPU to the Windows qube and the system “sees” it, can install nVidia drivers for it but it never starts and comes up with a Code: 43 error.
I’ve read in various external guides that newer GPUs now need to have their vbios edited otherwise they will not start in a VM. I did dump the vbios via GPU-Z, used hex edit and placed the edited file in /usr/share/vgabios/GB203.rom in Dom0. Now I’ve added my gpu to the windows-10.xml using this method: Custom libvirt config — core-admin mm_71b5ab21-0-g71b5ab2 documentation.
The output of /etc/qubes/templates/libvirt/xen/by-name/windows-10.xml is
{% block devices %}
{{ super() }}
<hostdev mode='subsystem' type='pci' managed='yes' nostrictreset='yes' permissive='yes'>
<source>
<address domain='0x0000' bus='0x01' slot='0x0' function='0x0'/>
</source>
<rom file='usr/share/vgabios/GB203.rom'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes' nostrictreset='yes' permissive='yes'>
<source>
<address domain='0x0000' bus='0x01' slot='0x0' function='0x1'/>
</source>
<rom file='usr/share/vgabios/GB203.rom'/>
</hostdev>
{% endblock %}
This works i.e. it attaches the pci devices to the qube and the rom entry exists in the windows-10.xml qubes recreates. But I’m not sure if the vbios is read because the results are the same as using qvm-pci a or the qube manager GUI. Will this load the vbios file from this location (Dom0) or does it need to be sent to the qube to be read? Does Qubes even support this libvirt feature? Thanks in advance for any answers.
It is weird that you need to edit you vbios. Multiple causes of Error 43 are described on the arch wiki page. Have you tried their solutions?
As far as I can tell vbios edit is only required on single gpu setups
Yeah, qubes uses libvirt, but it is xen under the hood, expect it to not behave exactly the same as libvirt on KVM.
Thanks for these resources!
The tool they recommend rom-parser gave me the output of:
Valid ROM signature found @36800h, PCIR offset 0h
Invalid PCIR signature: U�
That’s for the original untampered rom. Same results with the roms downloaded from TechPowerUp site.
The modded stripped version I am using shows this:
Valid ROM signature found @0h, PCIR offset ca0h
PCIR: type 0 (x86 PC-AT), vendor: 10de, device: 2c02, class: 030000
PCIR: revision 0, vendor revision: 1
Valid ROM signature found @fa00h, PCIR offset 520h
PCIR: type 3 (EFI), vendor: 0000, device: 0000, class: 000000
PCIR: revision 3, vendor revision: 0
EFI: Signature Valid, Subsystem: Boot, Machine: X64
Last image
Seems ok but these 0000 entries bother me. After loading the rom I seem to have gotten a bit farther inside Windows. When installing nVidia drivers it prompts me to reboot which it did not before and there is no more “device not started” message in the driver event log. The code 43 message is still present. I have disabled resizable BAR in the bios but still no success. One other thing I’ve noticed is that permissive='yes' setting does not get set in the windows-10.xml. Only nostrictreset='yes' is present.
Some more trials in trying to make nVidia drivers start.
I’ve been trying several of the solution from that arch wiki page.
I have added some legacy solutions to the VM’s xml file:
{% extends 'libvirt/xen.xml' %}
{% block features %}
{{ super() }}
<hyperv mode='custom'>
<vendor_id state='on' value='1234567890ab'/>
</hyperv>
<kvm>
<hidden state='on'/>
</kvm>
<ioapic driver='kvm'/>
{% endblock %}
{% block cpu %}
{{ super() }}
<feature policy='disable' name='amd-stibp'/>
{% endblock %}
{% block devices %}
{{ super() }}
<hostdev mode='subsystem' type='pci' managed='yes' nostrictreset='yes' permissive='yes'>
<source writeFiltering='no'>
<address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</source>
<rom file='/usr/share/vgabios/GB203.rom'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes' nostrictreset='yes' permissive='yes'>
<source writeFiltering='no'>
<address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
</source>
</hostdev>
{% endblock %}
I have no idea if they do anything on Qubes tbh.
I’m testing more with the BAR settings right now. I wonder if virtual bar is the culprit for code 43 in this case. From the arch wiki page I’ve tried some methods to manipulate it. Disabling resizable BAR in bios sets both physical and virtual BAR to 256MB. Enabling it sets physical BAR to 16GB.
This is the only method to change physical BAR size that worked:
sudo sh -c 'echo "0000:01:00.0" > /sys/bus/pci/devices/0000:01:00.0/driver/unbind'
sudo sh -c 'echo 14 > /sys/bus/pci/devices/0000:01:00.0/resource1_resize'
sudo sh -c 'echo "0000:01:00.0" > /sys/bus/pci/drivers/pciback/bind'
However only resource1_resize is available. There is no other *_resize. Physical BAR size can be set as low as 64MB but I cannot change the virtual BAR which persists on 256MB - in the wiki page they want it at 8MB. Any ideas welcome…