HVM no bootable device after gpu passthrough

ezar2 · June 8, 2022, 9:00am

Hello

I have created linux hvm with gpu (nvidia) passthrough everything looks ok except that after starting hvm I get an error:

Boot failed: could not read the boot disk
No bootable device

without nvidia gpu hvm boot fine
what have i changed:

- sudo nano /etc/default/grub
- GRUB_CMDLINE_LINUX=" ...  iommu=1 xen-pciback.hide=(01:00.0) rd.qubes.hide_pci=01:00.0 xen-pciback.passthrough=1 xen-pciback.permissive"
- sudo grub2-mkconfig -o /boot/grub2/grub.cfg
- sudo grub2-mkconfig -o /boot/efi/EFI/qubes/grub.cfg
- reboot
- cat  /sys/module/xen_pciback/parameters/hide
(01:00.0)
- cat /sys/module/xen_pciback/parameters/passthrough
Y
- cat /sys/modules/xen_pciback/parameters/permissive
Y
- qvm-pci a HVM_NAME --persistent -o permissive=true -o no-strict-reset=true

and I have not changed stubdom.

→ What could be causing this?

logs -> /var/log/xen/console/guest-game2-dm.log

[details="/var/log/xen/console/guest-game2-dm.log"]
[2022-06-08 04:13:59] Logfile Opened
[2022-06-08 04:14:00] Linux version 5.10.105-xen-stubdom (mockbuild@0c9ad8f2058f40c49bc934dcc2ea73c7) (gcc (GCC) 10.3.1 20210422 (Red Hat 10.3.1-1), GNU ld version 2.34-6.fc32) #1 Fri Apr 22 17:47:07 CEST 2022
[2022-06-08 04:14:00] pcifront pci-0: Creating PCI Frontend Bus 0000:01
[2022-06-08 04:14:00] pcifront pci-0: PCI host bridge to bus 0000:01
[2022-06-08 04:14:00] pci_bus 0000:01: root bus resource [io  0x0000-0xffff]
[2022-06-08 04:14:00] pci_bus 0000:01: root bus resource [mem 0x00000000-0x7fffffffff]
[2022-06-08 04:14:00] pci_bus 0000:01: root bus resource [bus 00-ff]
[2022-06-08 04:14:00] pci 0000:01:00.0: [10de:1c8d] type 00 class 0x030200
[2022-06-08 04:14:00] pci 0000:01:00.0: reg 0x10: [mem 0xa3000000-0xa3ffffff]
[2022-06-08 04:14:00] pci 0000:01:00.0: reg 0x14: [mem 0x90000000-0x9fffffff 64bit pref]
[2022-06-08 04:14:00] pci 0000:01:00.0: reg 0x1c: [mem 0xa0000000-0xa1ffffff 64bit pref]
[2022-06-08 04:14:00] pci 0000:01:00.0: reg 0x24: [io  0x4000-0x407f]
[2022-06-08 04:14:00] pcifront pci-0: claiming resource 0000:01:00.0/0
[2022-06-08 04:14:00] pcifront pci-0: claiming resource 0000:01:00.0/1
[2022-06-08 04:14:00] pcifront pci-0: claiming resource 0000:01:00.0/3
[2022-06-08 04:14:00] pcifront pci-0: claiming resource 0000:01:00.0/5
[2022-06-08 04:14:00] xen_netfront: backend supports XDP headroom
[2022-06-08 04:14:00] blkfront: xvda: flush diskcache: enabled; persistent grants: enabled; indirect descriptors: enabled;
[2022-06-08 04:14:00]  xvda: xvda1
[2022-06-08 04:14:00] blkfront: xvdb: flush diskcache: enabled; persistent grants: enabled; indirect descriptors: enabled;
[2022-06-08 04:14:00] blkfront: xvdc: flush diskcache: enabled; persistent grants: enabled; indirect descriptors: enabled;
[2022-06-08 04:14:00] Freeing unused kernel image (initmem) memory: 696K
[2022-06-08 04:14:00] Write protecting the kernel read-only data: 10240k
[2022-06-08 04:14:00] Freeing unused kernel image (text/rodata gap) memory: 2044K
[2022-06-08 04:14:00] Freeing unused kernel image (rodata/data gap) memory: 1648K
[2022-06-08 04:14:00] Run /init as init process
[2022-06-08 04:14:00] + mount -t devtmpfs none /dev
[2022-06-08 04:14:00] + mount -t sysfs /sys /sys
[2022-06-08 04:14:00] + mount -t proc /proc /proc
[2022-06-08 04:14:00] + mount -t tmpfs -o 'size=1m,nodev,noexec' /tmp /tmp
[2022-06-08 04:14:00] + mount -o remount,ro /
[2022-06-08 04:14:00] + echo 1
[2022-06-08 04:14:00] + printf '%d\n' 1073741824
[2022-06-08 04:14:00] + /bin/xenstore-read target
[2022-06-08 04:14:00] + domid=11
[2022-06-08 04:14:00] + xenstore-read /local/domain/11/vm
[2022-06-08 04:14:00] + vm_path=/vm/d8b513d9-cb26-48c8-b448-474a2539dd1c
[2022-06-08 04:14:00] + sort
[2022-06-08 04:14:00] + xenstore-list -p /vm/d8b513d9-cb26-48c8-b448-474a2539dd1c/image/dm-argv
[2022-06-08 04:14:00] + xenstore-read /vm/d8b513d9-cb26-48c8-b448-474a2539dd1c/image/dm-argv/001 /vm/d8b513d9-cb26-48c8-b448-474a2539dd1c/image/dm-argv/002 /vm/d8b513d9-cb26-48c8-b448-474a2539dd1c/image/dm-argv/003 /vm/d8b513d9-cb26-48c8-b448-474a2539dd1c/image/dm-argv/004 /vm/d8b513d9-cb26-48c8-b448-474a2539dd1c/image/dm-argv/005 /vm/d8b513d9-cb26-48c8-b448-474a2539dd1c/image/dm-argv/006 /vm/d8b513d9-cb26-48c8-b448-474a2539dd1c/image/dm-argv/007 /vm/d8b513d9-cb26-48c8-b448-474a2539dd1c/image/dm-argv/008 /vm/d8b513d9-cb26-48c8-b448-474a2539dd1c/image/dm-argv/009 /vm/d8b513d9-cb26-48c8-b448-474a2539dd1c/image/dm-argv/010 /vm/d8b513d9-cb26-48c8-b448-474a2539dd1c/image/dm-argv/011 /vm/d8b513d9-cb26-48c8-b448-474a2539dd1c/image/dm-argv/012 /vm/d8b513d9-cb26-48c8-b448-474a2539dd1c/image/dm-argv/013 /vm/d8b513d9-cb26-48c8-b448-474a2539dd1c/image/dm-argv/014 /vm/d8b513d9-cb26-48c8-b448-474a2539dd1c/image/dm-argv/015 /vm/d8b513d9-cb26-48c8-b448-474a2539dd1c/image/dm-argv/016 /vm/d8b513d9-cb26-48c8-b448-474a2539dd1c/image/dm-argv/017 /vm/d8b513d9-cb26-48c8-b448-474a2539dd1c/image/dm-argv/018 /vm/d8b513d9-cb26-48c8-b448-474a2539dd1c/image/dm-argv/019 /vm/d8b513d9-cb26-48c8-b448-474a2539dd1c/image/dm-argv/020 /vm/d8b513d9-cb26-48c8-b448-474a2539dd1c/image/dm-argv/021 /vm/d8b513d9-cb26-48c8-b448-474a2539dd1c/image/dm-argv/022 /vm/d8b513d9-cb26-48c8-b448-474a2539dd1c/image/dm-argv/023 /vm/d8b513d9-cb26-48c8-b448-474a2539dd1c/image/dm-argv/024 /vm/d8b513d9-cb26-48c8-b448-474a2539dd1c/image/dm-argv/025 /vm/d8b513d9-cb26-48c8-b448-474a2539dd1c/image/dm-argv/026 /vm/d8b513d9-cb26-48c8-b448-474a2539dd1c/image/dm-argv/027 /vm/d8b513d9-cb26-48c8-b448-474a2539dd1c/image/dm-argv/028 /vm/d8b513d9-cb26-48c8-b448-474a2539dd1c/image/dm-argv/029 /vm/d8b513d9-cb26-48c8-b448-474a2539dd1c/image/dm-argv/030 /vm/d8b513d9-cb26-48c8-b448-474a2539dd1c/image/dm-argv/031 /vm/d8b513d9-cb26-48c8-b448-474a2539dd1c/image/dm-argv/032 /vm/d8b513d9-cb26-48c8-b448-474a2539dd1c/image/dm-argv/033 /vm/d8b513d9-cb26-48c8-b448-474a2539dd1c/image/dm-argv/034 /vm/d8b513d9-cb26-48c8-b448-474a2539dd1c/image/dm-argv/035 /vm/d8b513d9-cb26-48c8-b448-474a2539dd1c/image/dm-argv/036
[2022-06-08 04:14:01] + dm_args='-xen-domid
[2022-06-08 04:14:01] 11
[2022-06-08 04:14:01] -no-shutdown
[2022-06-08 04:14:01] -nodefaults
[2022-06-08 04:14:01] -no-user-config
[2022-06-08 04:14:01] -name
[2022-06-08 04:14:01] game2
[2022-06-08 04:14:01] -display
[2022-06-08 04:14:01] none
[2022-06-08 04:14:01] -device
[2022-06-08 04:14:01] VGA,vgamem_mb=16
[2022-06-08 04:14:01] -boot
[2022-06-08 04:14:01] order=dc
[2022-06-08 04:14:01] -device
[2022-06-08 04:14:01] usb-ehci,id=ehci
[2022-06-08 04:14:01] -device
[2022-06-08 04:14:01] usb-tablet,bus=ehci.0
[2022-06-08 04:14:01] -smp
[2022-06-08 04:14:01] 4,maxcpus=4
[2022-06-08 04:14:01] -device
[2022-06-08 04:14:01] rtl8139,id=nic0,netdev=net0,mac=00:16:3e:5e:6c:00
[2022-06-08 04:14:01] -netdev
[2022-06-08 04:14:01] type=tap,id=net0,ifname=vif11.0-emu,script=no,downscript=no
[2022-06-08 04:14:01] -display
[2022-06-08 04:14:01] qubes-gui,domid=0,log-level=0
[2022-06-08 04:14:01] -machine
[2022-06-08 04:14:01] xenfv
[2022-06-08 04:14:01] -qubes-net:client_ip=10.137.0.18,dns_0=10.139.1.1,dns_1=10.139.1.2,gw=10.138.4.221,netmask=255.255.255.255
[2022-06-08 04:14:01] -m
[2022-06-08 04:14:01] 2484
[2022-06-08 04:14:01] -drive
[2022-06-08 04:14:01] file=/dev/xvda,if=ide,index=0,media=disk,format=host_device,cache=writeback
[2022-06-08 04:14:01] -drive
[2022-06-08 04:14:01] file=/dev/xvdb,if=ide,index=1,media=disk,format=host_device,cache=writeback
[2022-06-08 04:14:01] -drive
[2022-06-08 04:14:01] file=/dev/xvdc,if=ide,index=2,media=disk,format=host_device,cache=writeback'
[2022-06-08 04:14:01] + usb_args=
[2022-06-08 04:14:01] + test -e /bin/qrexec-agent
[2022-06-08 04:14:01] + sed -n '/^-soundhw/ {n;p}'
[2022-06-08 04:14:01] + echo '-xen-domid
[2022-06-08 04:14:01] 11
[2022-06-08 04:14:01] -no-shutdown
[2022-06-08 04:14:01] -nodefaults
[2022-06-08 04:14:01] -no-user-config
[2022-06-08 04:14:01] -name
[2022-06-08 04:14:01] game2
[2022-06-08 04:14:01] -display
[2022-06-08 04:14:01] none
[2022-06-08 04:14:01] -device
[2022-06-08 04:14:01] VGA,vgamem_mb=16
[2022-06-08 04:14:01] -boot
[2022-06-08 04:14:01] order=dc
[2022-06-08 04:14:01] -device
[2022-06-08 04:14:01] usb-ehci,id=ehci
[2022-06-08 04:14:01] -device
[2022-06-08 04:14:01] usb-tablet,bus=ehci.0
[2022-06-08 04:14:01] -smp
[2022-06-08 04:14:01] 4,maxcpus=4
[2022-06-08 04:14:01] -device
[2022-06-08 04:14:01] rtl8139,id=nic0,netdev=net0,mac=00:16:3e:5e:6c:00
[2022-06-08 04:14:01] -netdev
[2022-06-08 04:14:01] type=tap,id=net0,ifname=vif11.0-emu,script=no,downscript=no
[2022-06-08 04:14:01] -display
[2022-06-08 04:14:01] qubes-gui,domid=0,log-level=0
[2022-06-08 04:14:01] -machine
[2022-06-08 04:14:01] xenfv
[2022-06-08 04:14:01] -qubes-net:client_ip=10.137.0.18,dns_0=10.139.1.1,dns_1=10.139.1.2,gw=10.138.4.221,netmask=255.255.255.255
[2022-06-08 04:14:01] -m
[2022-06-08 04:14:01] 2484
[2022-06-08 04:14:01] -drive
[2022-06-08 04:14:01] file=/dev/xvda,if=ide,index=0,media=disk,format=host_device,cache=writeback
[2022-06-08 04:14:01] -drive
[2022-06-08 04:14:01] file=/dev/xvdb,if=ide,index=1,media=disk,format=host_device,cache=writeback
[2022-06-08 04:14:01] -drive
[2022-06-08 04:14:01] file=/dev/xvdc,if=ide,index=2,media=disk,format=host_device,cache=writeback'
[2022-06-08 04:14:01] + audio_model=
[2022-06-08 04:14:01] + '[' -n  ]
[2022-06-08 04:14:01] + sed -n /^-qubes-net:/p
[2022-06-08 04:14:01] + echo '-xen-domid
[2022-06-08 04:14:01] 11
[2022-06-08 04:14:01] -no-shutdown
[2022-06-08 04:14:01] -nodefaults
[2022-06-08 04:14:01] -no-user-config
[2022-06-08 04:14:01] -name
[2022-06-08 04:14:01] game2
[2022-06-08 04:14:01] -display
[2022-06-08 04:14:01] none
[2022-06-08 04:14:01] -device
[2022-06-08 04:14:01] VGA,vgamem_mb=16
[2022-06-08 04:14:01] -boot
[2022-06-08 04:14:01] order=dc
[2022-06-08 04:14:01] -device
[2022-06-08 04:14:01] usb-ehci,id=ehci
[2022-06-08 04:14:01] -device
[2022-06-08 04:14:01] usb-tablet,bus=ehci.0
[2022-06-08 04:14:01] -smp
[2022-06-08 04:14:01] 4,maxcpus=4
[2022-06-08 04:14:01] -device
[2022-06-08 04:14:01] rtl8139,id=nic0,netdev=net0,mac=00:16:3e:5e:6c:00
[2022-06-08 04:14:01] -netdev
[2022-06-08 04:14:01] type=tap,id=net0,ifname=vif11.0-emu,script=no,downscript=no
[2022-06-08 04:14:01] -display
[2022-06-08 04:14:01] qubes-gui,domid=0,log-level=0
[2022-06-08 04:14:01] -machine
[2022-06-08 04:14:01] xenfv
[2022-06-08 04:14:01] -qubes-net:client_ip=10.137.0.18,dns_0=10.139.1.1,dns_1=10.139.1.2,gw=10.138.4.221,netmask=255.255.255.255
[2022-06-08 04:14:01] -m
[2022-06-08 04:14:01] 2484
[2022-06-08 04:14:01] -drive
[2022-06-08 04:14:01] file=/dev/xvda,if=ide,index=0,media=disk,format=host_device,cache=writeback
[2022-06-08 04:14:01] -drive
[2022-06-08 04:14:01] file=/dev/xvdb,if=ide,index=1,media=disk,format=host_device,cache=writeback
[2022-06-08 04:14:01] -drive
[2022-06-08 04:14:01] file=/dev/xvdc,if=ide,index=2,media=disk,format=host_device,cache=writeback'
[2022-06-08 04:14:01] + net_args='-qubes-net:client_ip=10.137.0.18,dns_0=10.139.1.1,dns_1=10.139.1.2,gw=10.138.4.221,netmask=255.255.255.255'
[2022-06-08 04:14:01] + sed /^-qubes-net:/d
[2022-06-08 04:14:01] + echo '-xen-domid
[2022-06-08 04:14:01] 11
[2022-06-08 04:14:01] -no-shutdown
[2022-06-08 04:14:01] -nodefaults
[2022-06-08 04:14:01] -no-user-config
[2022-06-08 04:14:01] -name
[2022-06-08 04:14:01] game2
[2022-06-08 04:14:01] -display
[2022-06-08 04:14:01] none
[2022-06-08 04:14:01] -device
[2022-06-08 04:14:01] VGA,vgamem_mb=16
[2022-06-08 04:14:01] -boot
[2022-06-08 04:14:01] order=dc
[2022-06-08 04:14:01] -device
[2022-06-08 04:14:01] usb-ehci,id=ehci
[2022-06-08 04:14:01] -device
[2022-06-08 04:14:01] usb-tablet,bus=ehci.0
[2022-06-08 04:14:01] -smp
[2022-06-08 04:14:01] 4,maxcpus=4
[2022-06-08 04:14:01] -device
[2022-06-08 04:14:01] rtl8139,id=nic0,netdev=net0,mac=00:16:3e:5e:6c:00
[2022-06-08 04:14:01] -netdev
[2022-06-08 04:14:01] type=tap,id=net0,ifname=vif11.0-emu,script=no,downscript=no
[2022-06-08 04:14:01] -display
[2022-06-08 04:14:01] qubes-gui,domid=0,log-level=0
[2022-06-08 04:14:01] -machine
[2022-06-08 04:14:01] xenfv
[2022-06-08 04:14:01] -qubes-net:client_ip=10.137.0.18,dns_0=10.139.1.1,dns_1=10.139.1.2,gw=10.138.4.221,netmask=255.255.255.255
[2022-06-08 04:14:01] -m
[2022-06-08 04:14:01] 2484
[2022-06-08 04:14:01] -drive
[2022-06-08 04:14:01] file=/dev/xvda,if=ide,index=0,media=disk,format=host_device,cache=writeback
[2022-06-08 04:14:01] -drive
[2022-06-08 04:14:01] file=/dev/xvdb,if=ide,index=1,media=disk,format=host_device,cache=writeback
[2022-06-08 04:14:01] -drive
[2022-06-08 04:14:01] file=/dev/xvdc,if=ide,index=2,media=disk,format=host_device,cache=writeback'
[2022-06-08 04:14:01] + dm_args='-xen-domid
[2022-06-08 04:14:01] 11
[2022-06-08 04:14:01] -no-shutdown
[2022-06-08 04:14:01] -nodefaults
[2022-06-08 04:14:01] -no-user-config
[2022-06-08 04:14:01] -name
[2022-06-08 04:14:01] game2
[2022-06-08 04:14:01] -display
[2022-06-08 04:14:01] none
[2022-06-08 04:14:01] -device
[2022-06-08 04:14:01] VGA,vgamem_mb=16
[2022-06-08 04:14:01] -boot
[2022-06-08 04:14:01] order=dc
[2022-06-08 04:14:01] -device
[2022-06-08 04:14:01] usb-ehci,id=ehci
[2022-06-08 04:14:01] -device
[2022-06-08 04:14:01] usb-tablet,bus=ehci.0
[2022-06-08 04:14:01] -smp
[2022-06-08 04:14:01] 4,maxcpus=4
[2022-06-08 04:14:01] -device
[2022-06-08 04:14:01] rtl8139,id=nic0,netdev=net0,mac=00:16:3e:5e:6c:00
[2022-06-08 04:14:01] -netdev
[2022-06-08 04:14:01] type=tap,id=net0,ifname=vif11.0-emu,script=no,downscript=no
[2022-06-08 04:14:01] -display
[2022-06-08 04:14:01] qubes-gui,domid=0,log-level=0
[2022-06-08 04:14:01] -machine
[2022-06-08 04:14:01] xenfv
[2022-06-08 04:14:01] -m
[2022-06-08 04:14:01] 2484
[2022-06-08 04:14:01] -drive
[2022-06-08 04:14:01] file=/dev/xvda,if=ide,index=0,media=disk,format=host_device,cache=writeback
[2022-06-08 04:14:01] -drive
[2022-06-08 04:14:01] file=/dev/xvdb,if=ide,index=1,media=disk,format=host_device,cache=writeback
[2022-06-08 04:14:01] -drive
[2022-06-08 04:14:01] file=/dev/xvdc,if=ide,index=2,media=disk,format=host_device,cache=writeback'
[2022-06-08 04:14:01] + test -e /sys/class/net/eth0
[2022-06-08 04:14:01] + ip link set eth0 address fe:ff:ff:ff:ff:fe
[2022-06-08 04:14:01] + ip addr flush eth0
[2022-06-08 04:14:01] + ip link set eth0 up
[2022-06-08 04:14:01] + brctl addbr br0
[2022-06-08 04:14:01] + brctl addif br0 eth0
[2022-06-08 04:14:01] br0: port 1(eth0) entered blocking state
[2022-06-08 04:14:01] br0: port 1(eth0) entered disabled state
[2022-06-08 04:14:01] device eth0 entered promiscuous mode
[2022-06-08 04:14:01] + ip link set br0 up
[2022-06-08 04:14:01] br0: port 1(eth0) entered blocking state
[2022-06-08 04:14:01] br0: port 1(eth0) entered forwarding state
[2022-06-08 04:14:01] + get_net_arg dns_0
[2022-06-08 04:14:01] + sed -n 's/^.*[:,]dns_0=\([^,]\+\).*$/\1/p'
[2022-06-08 04:14:01] + echo '-qubes-net:client_ip=10.137.0.18,dns_0=10.139.1.1,dns_1=10.139.1.2,gw=10.138.4.221,netmask=255.255.255.255'
[2022-06-08 04:14:01] + net_dns_0=10.139.1.1
[2022-06-08 04:14:01] + get_net_arg dns_1
[2022-06-08 04:14:01] + sed -n 's/^.*[:,]dns_1=\([^,]\+\).*$/\1/p'
[2022-06-08 04:14:01] + echo '-qubes-net:client_ip=10.137.0.18,dns_0=10.139.1.1,dns_1=10.139.1.2,gw=10.138.4.221,netmask=255.255.255.255'
[2022-06-08 04:14:01] + net_dns_1=10.139.1.2
[2022-06-08 04:14:01] + get_net_arg client_ip
[2022-06-08 04:14:01] + sed -n 's/^.*[:,]client_ip=\([^,]\+\).*$/\1/p'
[2022-06-08 04:14:01] + echo '-qubes-net:client_ip=10.137.0.18,dns_0=10.139.1.1,dns_1=10.139.1.2,gw=10.138.4.221,netmask=255.255.255.255'
[2022-06-08 04:14:01] + net_client_ip=10.137.0.18
[2022-06-08 04:14:01] + get_net_arg netmask
[2022-06-08 04:14:01] + sed -n 's/^.*[:,]netmask=\([^,]\+\).*$/\1/p'
[2022-06-08 04:14:01] + echo '-qubes-net:client_ip=10.137.0.18,dns_0=10.139.1.1,dns_1=10.139.1.2,gw=10.138.4.221,netmask=255.255.255.255'
[2022-06-08 04:14:01] + net_netmask=255.255.255.255
[2022-06-08 04:14:01] + get_net_arg gw
[2022-06-08 04:14:01] + sed -n 's/^.*[:,]gw=\([^,]\+\).*$/\1/p'
[2022-06-08 04:14:01] + echo '-qubes-net:client_ip=10.137.0.18,dns_0=10.139.1.1,dns_1=10.139.1.2,gw=10.138.4.221,netmask=255.255.255.255'
[2022-06-08 04:14:01] + net_gw=10.138.4.221
[2022-06-08 04:14:01] + xenstore-read target
[2022-06-08 04:14:01] + udhcpd -f -I 10.138.4.221 -
[2022-06-08 04:14:01] + target=11
[2022-06-08 04:14:01] + device_model=device-model/11
[2022-06-08 04:14:01] + mkdir /tmp/qmp
[2022-06-08 04:14:01] udhcpd: started, v1.31.1
[2022-06-08 04:14:01] + kernel=
[2022-06-08 04:14:01] + grep -q ^-append
[2022-06-08 04:14:01] + echo '-xen-domid
[2022-06-08 04:14:01] 11
[2022-06-08 04:14:01] -no-shutdown
[2022-06-08 04:14:01] -nodefaults
[2022-06-08 04:14:01] -no-user-config
[2022-06-08 04:14:01] -name
[2022-06-08 04:14:01] game2
[2022-06-08 04:14:01] -display
[2022-06-08 04:14:01] none
[2022-06-08 04:14:01] -device
[2022-06-08 04:14:01] VGA,vgamem_mb=16
[2022-06-08 04:14:01] -boot
[2022-06-08 04:14:01] order=dc
[2022-06-08 04:14:01] -device
[2022-06-08 04:14:01] usb-ehci,id=ehci
[2022-06-08 04:14:01] -device
[2022-06-08 04:14:01] usb-tablet,bus=ehci.0
[2022-06-08 04:14:01] -smp
[2022-06-08 04:14:01] 4,maxcpus=4
[2022-06-08 04:14:01] -device
[2022-06-08 04:14:01] rtl8139,id=nic0,netdev=net0,mac=00:16:3e:5e:6c:00
[2022-06-08 04:14:01] -netdev
[2022-06-08 04:14:01] type=tap,id=net0,ifname=vif11.0-emu,script=no,downscript=no
[2022-06-08 04:14:01] -display
[2022-06-08 04:14:01] qubes-gui,domid=0,log-level=0
[2022-06-08 04:14:01] -machine
[2022-06-08 04:14:01] xenfv
[2022-06-08 04:14:01] -m
[2022-06-08 04:14:01] 2484
[2022-06-08 04:14:01] -drive
[2022-06-08 04:14:01] file=/dev/xvda,if=ide,index=0,media=disk,format=host_device,cache=writeback
[2022-06-08 04:14:01] -drive
[2022-06-08 04:14:01] file=/dev/xvdb,if=ide,index=1,media=disk,format=host_device,cache=writeback
[2022-06-08 04:14:01] -drive
[2022-06-08 04:14:01] file=/dev/xvdc,if=ide,index=2,media=disk,format=host_device,cache=writeback'
[2022-06-08 04:14:01] + mkfifo /tmp/qmp/qemu.in /tmp/qmp/qemu.out
[2022-06-08 04:14:01] + set +x
[2022-06-08 04:14:01] Clearing kmsg buffer...
[2022-06-08 04:14:01] + set +x
[2022-06-08 04:14:01] + set +x
[2022-06-08 04:14:01] + sed 's/\$STUBDOM_RESTORE_INCOMING_ARG/fd:3/'
[2022-06-08 04:14:01] + echo '-xen-domid
[2022-06-08 04:14:01] 11
[2022-06-08 04:14:01] -no-shutdown
[2022-06-08 04:14:01] -nodefaults
[2022-06-08 04:14:01] -no-user-config
[2022-06-08 04:14:01] -name
[2022-06-08 04:14:01] game2
[2022-06-08 04:14:01] -display
[2022-06-08 04:14:01] none
[2022-06-08 04:14:01] -device
[2022-06-08 04:14:01] VGA,vgamem_mb=16
[2022-06-08 04:14:01] -boot
[2022-06-08 04:14:01] order=dc
[2022-06-08 04:14:01] -device
[2022-06-08 04:14:01] usb-ehci,id=ehci
[2022-06-08 04:14:01] -device
[2022-06-08 04:14:01] usb-tablet,bus=ehci.0
[2022-06-08 04:14:01] -smp
[2022-06-08 04:14:01] 4,maxcpus=4
[2022-06-08 04:14:01] -device
[2022-06-08 04:14:01] rtl8139,id=nic0,netdev=net0,mac=00:16:3e:5e:6c:00
[2022-06-08 04:14:01] -netdev
[2022-06-08 04:14:01] type=tap,id=net0,ifname=vif11.0-emu,script=no,downscript=no
[2022-06-08 04:14:01] -display
[2022-06-08 04:14:01] qubes-gui,domid=0,log-level=0
[2022-06-08 04:14:01] -machine
[2022-06-08 04:14:01] xenfv
[2022-06-08 04:14:01] -m
[2022-06-08 04:14:01] 2484
[2022-06-08 04:14:01] -drive
[2022-06-08 04:14:01] file=/dev/xvda,if=ide,index=0,media=disk,format=host_device,cache=writeback
[2022-06-08 04:14:01] -drive
[2022-06-08 04:14:01] file=/dev/xvdb,if=ide,index=1,media=disk,format=host_device,cache=writeback
[2022-06-08 04:14:01] -drive
[2022-06-08 04:14:01] file=/dev/xvdc,if=ide,index=2,media=disk,format=host_device,cache=writeback'
[2022-06-08 04:14:01] + dm_args='-xen-domid
[2022-06-08 04:14:01] 11
[2022-06-08 04:14:01] -no-shutdown
[2022-06-08 04:14:01] -nodefaults
[2022-06-08 04:14:01] -no-user-config
[2022-06-08 04:14:01] -name
[2022-06-08 04:14:01] game2
[2022-06-08 04:14:01] -display
[2022-06-08 04:14:01] none
[2022-06-08 04:14:01] -device
[2022-06-08 04:14:01] VGA,vgamem_mb=16
[2022-06-08 04:14:01] -boot
[2022-06-08 04:14:01] order=dc
[2022-06-08 04:14:01] -device
[2022-06-08 04:14:01] usb-ehci,id=ehci
[2022-06-08 04:14:01] -device
[2022-06-08 04:14:01] usb-tablet,bus=ehci.0
[2022-06-08 04:14:01] -smp
[2022-06-08 04:14:01] 4,maxcpus=4
[2022-06-08 04:14:01] -device
[2022-06-08 04:14:01] rtl8139,id=nic0,netdev=net0,mac=00:16:3e:5e:6c:00
[2022-06-08 04:14:01] -netdev
[2022-06-08 04:14:01] type=tap,id=net0,ifname=vif11.0-emu,script=no,downscript=no
[2022-06-08 04:14:01] -display
[2022-06-08 04:14:01] qubes-gui,domid=0,log-level=0
[2022-06-08 04:14:01] -machine
[2022-06-08 04:14:01] xenfv
[2022-06-08 04:14:01] -m
[2022-06-08 04:14:01] 2484
[2022-06-08 04:14:01] -drive
[2022-06-08 04:14:01] file=/dev/xvda,if=ide,index=0,media=disk,format=host_device,cache=writeback
[2022-06-08 04:14:01] -drive
[2022-06-08 04:14:01] file=/dev/xvdb,if=ide,index=1,media=disk,format=host_device,cache=writeback
[2022-06-08 04:14:01] -drive
[2022-06-08 04:14:01] file=/dev/xvdc,if=ide,index=2,media=disk,format=host_device,cache=writeback'
[2022-06-08 04:14:01] + xenstore-read device/console/2
[2022-06-08 04:14:01] + xenstore-read device/console/1
[2022-06-08 04:14:01] + IFS='
[2022-06-08 04:14:01] '
[2022-06-08 04:14:01] + set -f
[2022-06-08 04:14:01] + set +f
[2022-06-08 04:14:01] + unset IFS
[2022-06-08 04:14:01] + qemu_pid=154
[2022-06-08 04:14:01] + '[' '!' -e /tmp/qemu.qmp ]
[2022-06-08 04:14:01] + sleep 0.1
[2022-06-08 04:14:01] + qemu -sandbox 'on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny' -chardev 'pipe,path=/tmp/qmp/qemu,id=m' -mon 'chardev=m,mode=control' -chardev 'socket,server,nowait,path=/tmp/qemu.qmp,id=m2' -mon 'chardev=m2,mode=control' -xen-domid 11 -no-shutdown -nodefaults -no-user-config -name game2 -display none -device 'VGA,vgamem_mb=16' -boot 'order=dc' -device 'usb-ehci,id=ehci' -device 'usb-tablet,bus=ehci.0' -smp '4,maxcpus=4' -device 'rtl8139,id=nic0,netdev=net0,mac=00:16:3e:5e:6c:00' -netdev 'type=tap,id=net0,ifname=vif11.0-emu,script=no,downscript=no' -display 'qubes-gui,domid=0,log-level=0' -machine xenfv -m 2484 -drive 'file=/dev/xvda,if=ide,index=0,media=disk,format=host_device,cache=writeback' -drive 'file=/dev/xvdb,if=ide,index=1,media=disk,format=host_device,cache=writeback' -drive 'file=/dev/xvdc,if=ide,index=2,media=disk,format=host_device,cache=writeback'
[2022-06-08 04:14:01] qemu: -chardev socket,server,nowait,path=/tmp/qemu.qmp,id=m2: warning: short-form boolean option 'server' deprecated
[2022-06-08 04:14:01] Please use server=on instead
[2022-06-08 04:14:01] qemu: -chardev socket,server,nowait,path=/tmp/qemu.qmp,id=m2: warning: short-form boolean option 'nowait' deprecated
[2022-06-08 04:14:01] Please use wait=off instead
[2022-06-08 04:14:01] random: qemu: uninitialized urandom read (16 bytes read)
[2022-06-08 04:14:01] + tee /proc/self/fd/2
[2022-06-08 04:14:01] + echo '{"execute": "qmp_capabilities"}'
[2022-06-08 04:14:01] {"QMP": {"version": {"qemu": {"micro": 0, "minor": 1, "major": 6}, "package": ""}, "capabilities": ["oob"]}}

[2022-06-08 04:14:01] {"execute": "qmp_capabilities"}
[2022-06-08 04:14:01] + '[' '!' -e /tmp/qemu.qmp ]
[2022-06-08 04:14:01] + '[' -e /proc/self/fd/4 ]
[2022-06-08 04:14:01] + '[' -e /proc/self/fd/3 ]
[2022-06-08 04:14:01] + true
[2022-06-08 04:14:01] + printf '==== Press enter for shell ====\n'
[2022-06-08 04:14:01] ==== Press enter for shell ====
[2022-06-08 04:14:01] + read
[2022-06-08 04:14:01] + vchan-socket-proxy 0 device-model/11/qmp-vchan /tmp/qemu.qmp
[2022-06-08 04:14:01] qubes_gui/init: 573
[2022-06-08 04:14:01] qubes_gui/init: 582
[2022-06-08 04:14:01] qubes_gui/init: 584
[2022-06-08 04:14:01] br0: port 2(vif11.0-emu) entered blocking state
[2022-06-08 04:14:01] br0: port 2(vif11.0-emu) entered disabled state
[2022-06-08 04:14:01] device vif11.0-emu entered promiscuous mode
[2022-06-08 04:14:01] br0: port 2(vif11.0-emu) entered blocking state
[2022-06-08 04:14:01] br0: port 2(vif11.0-emu) entered forwarding state
[2022-06-08 04:14:01] qubes_gui/init[611]: version sent, waiting for xorg conf
[2022-06-08 04:14:01] {"return": {}}

[2022-06-08 04:14:01] written 110 bytes to vchan
[2022-06-08 04:14:01] written 34 bytes to vchan
[2022-06-08 04:14:01] [00:06.0] xen_pt_realize: Assigning real physical device 01:00.0 to devfn 0x30
[2022-06-08 04:14:01] [00:06.0] xen_pt_register_regions: IO region 0 registered (size=0x01000000 base_addr=0xa3000000 type: 0x0)
[2022-06-08 04:14:01] [00:06.0] xen_pt_register_regions: IO region 1 registered (size=0x10000000 base_addr=0x90000000 type: 0x4)
[2022-06-08 04:14:01] [00:06.0] xen_pt_register_regions: IO region 3 registered (size=0x02000000 base_addr=0xa0000000 type: 0x4)
[2022-06-08 04:14:01] [00:06.0] xen_pt_register_regions: IO region 5 registered (size=0x00000080 base_addr=0x00004000 type: 0x1)
[2022-06-08 04:14:01] [00:06.0] xen_pt_config_reg_init: Offset 0x000e mismatch! Emulated=0x0080, host=0x0000, syncing to 0x0000.
[2022-06-08 04:14:01] [00:06.0] xen_pt_config_reg_init: Offset 0x0010 mismatch! Emulated=0x0000, host=0xa3000000, syncing to 0xa3000000.
[2022-06-08 04:14:01] [00:06.0] xen_pt_config_reg_init: Offset 0x0014 mismatch! Emulated=0x0000, host=0x9000000c, syncing to 0x9000000c.
[2022-06-08 04:14:01] [00:06.0] xen_pt_config_reg_init: Offset 0x001c mismatch! Emulated=0x0000, host=0xa000000c, syncing to 0xa000000c.
[2022-06-08 04:14:01] [00:06.0] xen_pt_config_reg_init: Offset 0x0024 mismatch! Emulated=0x0000, host=0x4001, syncing to 0x4001.
[2022-06-08 04:14:01] [00:06.0] xen_pt_config_reg_init: Offset 0x0062 mismatch! Emulated=0x0000, host=0x0003, syncing to 0x0003.
[2022-06-08 04:14:01] [00:06.0] xen_pt_config_reg_init: Offset 0x006a mismatch! Emulated=0x0000, host=0x0080, syncing to 0x0080.
[2022-06-08 04:14:01] [00:06.0] xen_pt_config_reg_init: Offset 0x007c mismatch! Emulated=0x0000, host=0x12c8de1, syncing to 0x12c8de1[2022-06-08 04:14:02] .
[2022-06-08 04:14:02] [00:06.0] xen_pt_config_reg_init: Offset 0x008a mismatch! Emulated=0x0000, host=0x1103, syncing to 0x1103.
[2022-06-08 04:14:02] [00:06.0] xen_pt_pci_intx: intx=1
[2022-06-08 04:14:02] [00:06.0] xen_pt_realize: Real physical device 01:00.0 registered successfully
[2022-06-08 04:14:02] written 34 bytes to vchan
[2022-06-08 04:14:02] written 2048 bytes to vchan
[2022-06-08 04:14:02] written 1149 bytes to vchan
[2022-06-08 04:14:02] written 110 bytes to vchan
[2022-06-08 04:14:02] written 34 bytes to vchan
[2022-06-08 04:14:02] written 34 bytes to vchan
[2022-06-08 04:14:02] qubes_gui/init[622]: got xorg conf, creating window
[2022-06-08 04:14:02] qubes_gui/init: 632
[2022-06-08 04:14:02] configure msg, x/y 640 325 (was 0 0), w/h 640 480
[2022-06-08 04:14:02] configure msg, x/y 640 325 (was 640 325), w/h 640 480
[2022-06-08 04:14:02] processing error - resetting ehci HC
[2022-06-08 04:14:03] random: crng init done


[/details]

tzwcfq · June 8, 2022, 9:20am

I had the same issue. You need to downgrade xen-hvm-stubdom-linux and xen-hvm-stubdom-linux-full to the previous version 1.2.3-1.fc32 and then apply patch from this guide:
Contents/windows-gaming-hvm.md at master · Qubes-Community/Contents · GitHub
AppVM with GPU pass-through crashes when more than 3.5 GB (3584MB) of RAM is assigned to it · Issue #4321 · QubesOS/qubes-issues · GitHub

ezar2 · June 8, 2022, 11:45am

Yes it was it.

Thank you

AbusiveHeathenry · July 30, 2022, 1:57pm

I had this error too after upgrading dom0 to qubes-dom0-current-testing, was able to fix it without downgrading, just need to reapply my previous changes to the init file in /usr/libexec/xen/boot/qemu-stubdom-linux-rootfs

Installed Packages
xen-hvm-stubdom-linux-full.x86_64                                                      1.2.4-1.fc32                                                       @qubes-dom0-cached

deeplow · January 9, 2024, 8:12pm

Supposedly with the Qubes 4.2 on testing with the latest Xen, the <3.5G issue should be fixed:

However, I have that patch but I am still getting a No bootable device. I should also note that this happens with any PCI-e device attached, not just a graphics card.

Any ideas?

apparatus · January 9, 2024, 8:33pm

Maybe you have old patched stubdom files or xen.xml?
Try to reinstall xen-hvm-stubdom-linux package to get the unpatched stubdom.

deeplow · January 9, 2024, 9:49pm

Unfortunately no. That was a clean install. Just in case I also tried the patch (with the appropriate modified package path) but that didn’t help.

Cameron · January 10, 2024, 12:41am

I have not done a clean install (upgraded from 4.1) but I am experiencing the same issue on this end. I have tried reinstalling both xen-hvm-stubdom-linux as well as xen-hvm-stubdom-linux-full.

Applying the patch with a number of variations (3.5G was previously working) has not yielded any results over here yet.

I have also downgraded both packages and applied the old stubdom patch just to test and no change.

neowutran · January 10, 2024, 7:58am

the recent patch applied to xen that should fix the max-ram-below-4g thing modified the behavior of this loop (allow_memory_relocate is now defined to 0 instead of 1, it should never have been set to 1 for our usecase )

github.com

xen-project/xen/blob/master/tools/firmware/hvmloader/pci.c#L353


      
           * if we're running qemu-traditional.  Items that don't fit will be
           * relocated into the 64-bit address space.
           *
           * This loop now does the following:
           * - If allow_memory_relocate, increase the MMIO hole until it's
           *   big enough, or until it's 2GiB
           * - If !allow_memory_relocate, increase the MMIO hole until it's
           *   big enough, or until it's 2GiB, or until it overlaps guest
           *   memory
           */
          while ( (mmio_total > (pci_mem_end - pci_mem_start))
                  && ((pci_mem_start << 1) != 0)
                  && (allow_memory_relocate
                      || (((pci_mem_start << 1) >> PAGE_SHIFT)
                          >= hvm_info->low_mem_pgend)) )
              pci_mem_start <<= 1;
          
          /*
           * Try to accommodate RMRRs in our MMIO region on a best-effort basis.
           * If we have RMRRs in the range, then make pci_mem_start just after
           * hvm_info->low_mem_pgend.

You could try to downgrade the xen version to before 4.17.2-8 to check if it work with the manual max-ram-below-4g workaround

In case everything else fail, someone could create a patch to add log in this code to monitor the value of all the important variables ( pci_mem_start / hvm_info->low_mem_pgend / hvm_info->low_mem_pgend << PAGE_SHIFT / pci_mem_end ), recompile xen with this patch, activate all xen logs, start the HVM and publish the logs related to the startup of the HVM. And compare it with the logs of people that doesn’t have this issue

Could be interesting to check the influence of this patch qemu: fix TOLUD for PCI passthrough by mati7337 · Pull Request #44 · QubesOS/qubes-vmm-xen-stubdom-linux · GitHub ( is it still needed ? does it have an influence on the issue you are having ? )

deeplow · January 10, 2024, 10:59am

Massive thanks. I will try it and report back.

elqub · January 10, 2024, 6:07pm

Would you be open to installing 4.2 rc3 and see if passthrough works with that version? It worked for me, but not the newer versions when installed from iso. 4.2 did work if I upgraded from rc3.

Cameron · January 12, 2024, 1:32am

Attempted to downgrade to 4.17.2-7 with sudo qubes-dom0-update --action=downgrade xen, then re-applied the stubdom patch and rebooted and the VM “boots”, however I get a bluescreen when booting the HVM seconds after the desktop loads, no other monitors initialize either. It boots fine with no PCI devices and changing gpu to nogpu in the name.

GPU-Z is only showing me 4 sensors as well which is odd for the brief moment that I am able to get in before it BSODs.

I have also tried downgrading Xen a second time (down to 2001:4.17.2-5.fc37), as well as downgrading both stubdom packages (to 4.2.7-1.fc37) and re-patching again to no avail.

Nobara displays the 3070 with lspci but no GPU shows in System Monitor and nvtop gives me “No GPU to Monitor”. Seems like it’s not loading properly. Here’s the output of lspci -vvv

Video card is excluded in Grub as it was in 4.1 with rd.qubes.hide_pci=0c:00.0,0c:00.1"

lspci on dom0:

0c:00.0 VGA compatible controller: NVIDIA Corporation GA104 [GeForce RTX 3070] (rev a1) (prog-if 00 [VGA controller])
	Subsystem: Micro-Star International Co., Ltd. [MSI] Device 3908
	Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin A routed to IRQ 54
	Region 0: Memory at fa000000 (32-bit, non-prefetchable) [size=16M]
	Region 1: Memory at b0000000 (64-bit, prefetchable) [size=256M]
	Region 3: Memory at c0000000 (64-bit, prefetchable) [size=32M]
	Region 5: I/O ports at f000 [size=128]
	Expansion ROM at 000c0000 [disabled] [size=128K]
	Capabilities: <access denied>
	Kernel driver in use: pciback
	Kernel modules: nouveau

0c:00.1 Audio device: NVIDIA Corporation GA104 High Definition Audio Controller (rev a1)
	Subsystem: Micro-Star International Co., Ltd. [MSI] Device 3908
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx+
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin B routed to IRQ 55
	Region 0: Memory at fb080000 (32-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: pciback
	Kernel modules: snd_hda_intel

Cameron · January 13, 2024, 7:40pm

Following up here, get the same issue on a clean 4.2 install with the following done:

Grub set to exclude my RTX 3070 and built.
Xorg conf created to load my AMD card as it’s in the secondary slot
Trying to boot any HVM with the 3070 attached fails with “No bootable device”.

I can boot it with my USB Controller, network card, and NVMe attached, however neither the USB Controller nor Network card function properly (no network connection, no USB connection)

EDIT:

Clean 4.1 install with the above done with the stubdom patch still works properly so going to stick with 4.1 for now since I only have the one system and need it for both work and gaming.

fore567 · March 1, 2024, 6:44pm

I get the same error on 4.2 any fixes?