AMD GPU passthrough to VM for gaming

I have been trying passing through AMD RX 580 GPU to VM(debian 11) for gaming, I have read multiple GPU passthrough threads on here, but none that I found to answer my question.

My dom0 uses intel’s iGPU.

I have successfully passed the VGA PCI device to VM and started it, I followed this tutorial: Qubes OS article

in addition to that tutorial, I edited dom0’s grub config, adding to /etc/default/grub:

GRUB_CMDLINE_LINUX="... rd.qubes.hide_pci=01:00.0 amd_iommu=on"

then I did: sudo grub2-mkconfig -o /boot/grub2/grub.cfg

but when I start xorg script from that tutorial, I do not know what to do next, it does not open any windows using the GPU, all I see is this in terminal:

sudo ./xorgX1.sh /usr/bin/openbox

X.Org X Server 1.20.11
X Protocol Version 11, Revision 0
Build Operating System: linux Debian
Current Operating System: Linux debian-11-gpu 5.10.0-20-amd64 #1 SMP Debian 5.10.158-2 (2022-12-13) x86_64
Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.10.0-20-amd64 root=/dev/mapper/dmroot ro xen_scrub_pages=0 root=/dev/mapper/dmroot console=tty0 console=hvc0 swiotlb=8192 noresume quiet
Build Date: 17 December 2022  10:00:08AM
xorg-server 2:1.20.11-1+deb11u4 (https://www.debian.org/support) 
Current version of pixman: 0.40.0
	Before reporting problems, check http://wiki.x.org
	to make sure that you have the latest version.
Markers: (--) probed, (**) from config file, (==) default setting,
	(++) from command line, (!!) notice, (II) informational,
	(WW) warning, (EE) error, (NI) not implemented, (??) unknown.
(==) Log file: "/var/log/Xorg.1.log", Time: Sat Jan 14 20:55:57 2023
(++) Using config file: "/home/user/AOC.conf"
(==) Using system config directory "/usr/share/X11/xorg.conf.d"
(II) AMDGPU(0): [KMS] Kernel modesetting enabled.

what am I missing? I do not know how to launch a program using the AMD GPU, it always launches them with software rendering instead.

can you post the result of the command “sudo dmesg” (launched from your gaming vm)

It was too long to post here, I’ve posted the result at: https://pastebin.com/raw/ydG98qEq

I noticed it sometimes showing “Obt-Message: Xinerama extension is not present on the server” after starting xorg server, which may be relevant, but I am not sure if it is. (I have only 1 monitor)

The interesting part is this one

[    2.929662] [drm] Initialized amdgpu 3.40.0 20150101 for 0000:00:06.0 on minor 1
[    4.392237] [drm] Fence fallback timer expired on ring gfx
[    4.904303] [drm] Fence fallback timer expired on ring comp_1.0.0
[    5.416237] [drm] Fence fallback timer expired on ring comp_1.1.0
[    5.928206] [drm] Fence fallback timer expired on ring comp_1.2.0
[    6.440223] [drm] Fence fallback timer expired on ring comp_1.3.0
[    6.952291] [drm] Fence fallback timer expired on ring comp_1.0.1
[    7.464239] [drm] Fence fallback timer expired on ring comp_1.1.1
[    7.980205] [drm] Fence fallback timer expired on ring comp_1.2.1
[    8.488291] [drm] Fence fallback timer expired on ring comp_1.3.1
[    9.000210] [drm] Fence fallback timer expired on ring sdma0
[    9.512236] [drm] Fence fallback timer expired on ring sdma1
[   10.056258] [drm] Fence fallback timer expired on ring uvd
[   10.568292] [drm] Fence fallback timer expired on ring uvd_enc0
[   11.080250] [drm] Fence fallback timer expired on ring uvd_enc1
[   11.688206] [drm] Fence fallback timer expired on ring vce0
[   12.712403] amdgpu 0000:00:06.0: [drm:amdgpu_ib_ring_tests [amdgpu]] *ERROR* IB test failed on vce0 (-110).
[   12.712466] [drm:process_one_work] *ERROR* ib ring test failed (-110).
[   13.224316] [drm] Fence fallback timer expired on ring sdma1
[   14.120315] [drm] Fence fallback timer expired on ring sdma0
[   24.936240] [drm] Fence fallback timer expired on ring sdma1
[   24.968239] [drm] Fence fallback timer expired on ring sdma0
[   25.448246] [drm] Fence fallback timer expired on ring sdma1
[   25.473961] amdgpu 0000:00:06.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=io+mem
[   25.480202] [drm] Fence fallback timer expired on ring sdma0

add “pci=nomsi” to your grub boot param and retry ( resend the result of “sudo dmesg” and “cat /proc/cmdline”

Thank you for the reply. I added pci=nomsi to dom0’s grub.
executed on gaming VM:
sudo dmesg: https://pastebin.com/raw/Zt4AqAye
cat /proc/cmdline:

BOOT_IMAGE=/boot/vmlinuz-5.10.0-20-amd64 root=/dev/mapper/dmroot ro xen_scrub_pages=0 root=/dev/mapper/dmroot console=tty0 console=hvc0 swiotlb=8192 noresume quiet

sorry, “pci=nomsi” in the guest grub boot param, not dom0

Added pci=nomsi to gaming VM’s grub.
sudo dmesg: https://pastebin.com/raw/ghBqiUUs
cat /proc/cmdline: BOOT_IMAGE=/boot/vmlinuz-5.10.0-20-amd64 root=/dev/mapper/dmroot ro pci=nomsi xen_scrub_pages=0 root=/dev/mapper/dmroot console=tty0 console=hvc0 swiotlb=8192 noresume quiet

The log look good now. You tried to launch xorg ?
What message do you get ? you have the screen connected to the rx 580 ready ?

Yes, I tried launching xorg, but nothing happens other than the usual text in the terminal. I have not connected rx 580 to monitor, I wasn’t sure if I have to, only iGPU is connected to monitor. I have a problem that if I connect both iGPU and rx580 to monitor, Qubes OS does not boot, only one at a time boots. Could it help to reinstall Qubes OS?

When you do this kind of passthrough (and want to display things), you need to have a monitor dedicated for this VM. So you need 2 monitor, or you can use a splitter if you don’t have/don’t want to have 2 monitor

Multimonitor would make things easier and he needs a video cable going out from that RX580 probably HDMI or display port ones going into a monitor.

Many Monitors have multiple inputs and he could switch to the other port via the Monitor Buttons.

Yes, I do have multiple inputs, but it does not seem to work when both are plugged into computer, I try to switch between them and it only goes back to the first screen, I am guessing it is caused by that during initial boot, with both plugged, it does not boot, so plugging it at runtime may not be accepted?

I noticed I get this in terminal on gaming VM when I plug in rx580 at runtime:

[../i3bar/src/xcb.c:1042] ERROR: PropertyNotify received for unknown window 00c00002

if I restart xorg server with rx580 plugged, I get this error:

sudo cat /var/log/Xorg.1.log
[  2079.447] 
X.Org X Server 1.20.11
X Protocol Version 11, Revision 0
[  2079.447] Build Operating System: linux Debian
[  2079.447] Current Operating System: Linux debian-11-gpu 5.10.0-20-amd64 #1 SMP Debian 5.10.158-2 (2022-12-13) x86_64
[  2079.447] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.10.0-20-amd64 root=/dev/mapper/dmroot ro pci=nomsi xen_scrub_pages=0 root=/dev/mapper/dmroot console=tty0 console=hvc0 swiotlb=8192 noresume quiet
[  2079.447] Build Date: 17 December 2022  10:00:08AM
[  2079.447] xorg-server 2:1.20.11-1+deb11u4 (https://www.debian.org/support) 
[  2079.447] Current version of pixman: 0.40.0
[  2079.448] 	Before reporting problems, check http://wiki.x.org
	to make sure that you have the latest version.
[  2079.448] Markers: (--) probed, (**) from config file, (==) default setting,
	(++) from command line, (!!) notice, (II) informational,
	(WW) warning, (EE) error, (NI) not implemented, (??) unknown.
[  2079.448] (==) Log file: "/var/log/Xorg.1.log", Time: Sun Jan 15 12:59:28 2023
[  2079.448] (++) Using config file: "/home/user/AOC.conf"
[  2079.448] (==) Using system config directory "/usr/share/X11/xorg.conf.d"
[  2079.448] (==) ServerLayout "Gaming"
[  2079.448] (**) |-->Screen "AMD AOC" (0)
[  2079.448] (**) |   |-->Monitor "AOC"
[  2079.448] (**) |   |-->Device "AMD"
[  2079.448] (==) Automatically adding devices
[  2079.448] (==) Automatically enabling devices
[  2079.448] (==) Automatically adding GPU devices
[  2079.448] (==) Max clients allowed: 256, resource mask: 0x1fffff
[  2079.448] (WW) The directory "/usr/share/fonts/X11/cyrillic" does not exist.
[  2079.448] 	Entry deleted from font path.
[  2079.448] (WW) The directory "/usr/share/fonts/X11/100dpi/" does not exist.
[  2079.448] 	Entry deleted from font path.
[  2079.448] (WW) The directory "/usr/share/fonts/X11/75dpi/" does not exist.
[  2079.448] 	Entry deleted from font path.
[  2079.448] (WW) The directory "/usr/share/fonts/X11/Type1" does not exist.
[  2079.448] 	Entry deleted from font path.
[  2079.448] (WW) The directory "/usr/share/fonts/X11/100dpi" does not exist.
[  2079.448] 	Entry deleted from font path.
[  2079.448] (WW) The directory "/usr/share/fonts/X11/75dpi" does not exist.
[  2079.448] 	Entry deleted from font path.
[  2079.448] (==) FontPath set to:
	/usr/share/fonts/X11/misc,
	built-ins
[  2079.448] (==) ModulePath set to "/usr/lib/xorg/modules"
[  2079.448] (II) The server relies on udev to provide the list of input devices.
	If no devices become available, reconfigure udev or disable AutoAddDevices.
[  2079.448] (II) Loader magic: 0x557a61f3de40
[  2079.448] (II) Module ABI versions:
[  2079.448] 	X.Org ANSI C Emulation: 0.4
[  2079.448] 	X.Org Video Driver: 24.1
[  2079.448] 	X.Org XInput driver : 24.1
[  2079.448] 	X.Org Server Extension : 10.0
[  2079.448] (--) using VT number 2

[  2079.448] (II) systemd-logind: logind integration requires -keeptty and -keeptty was not provided, disabling logind integration
[  2079.449] (II) xfree86: Adding drm device (/dev/dri/card0)
[  2079.449] (II) xfree86: Adding drm device (/dev/dri/card1)
[  2079.460] (--) PCI:*(0@0:3:0) 1234:1111:1af4:1100 rev 2, Mem @ 0xf1000000/16777216, 0xf2270000/4096, BIOS @ 0x????????/131072
[  2079.460] (--) PCI: (0@0:6:0) 1002:67df:1787:d580 rev 231, Mem @ 0xe0000000/268435456, 0xf2000000/2097152, 0xf2200000/262144, I/O @ 0x0000c200/256, BIOS @ 0x????????/131072
[  2079.460] (II) LoadModule: "glx"
[  2079.460] (II) Loading /usr/lib/xorg/modules/extensions/libglx.so
[  2079.461] (II) Module glx: vendor="X.Org Foundation"
[  2079.461] 	compiled for 1.20.11, module version = 1.0.0
[  2079.461] 	ABI class: X.Org Server Extension, version 10.0
[  2079.461] (II) LoadModule: "amdgpu"
[  2079.461] (II) Loading /usr/lib/xorg/modules/drivers/amdgpu_drv.so
[  2079.461] (II) Module amdgpu: vendor="X.Org Foundation"
[  2079.461] 	compiled for 1.20.9, module version = 19.1.0
[  2079.461] 	Module class: X.Org Video Driver
[  2079.461] 	ABI class: X.Org Video Driver, version 24.1
[  2079.461] (II) AMDGPU: Driver for AMD Radeon:
	All GPUs supported by the amdgpu kernel driver
[  2079.463] (II) AMDGPU(0): [KMS] Kernel modesetting enabled.
[  2079.464] (II) AMDGPU(0): Creating default Display subsection in Screen section
	"AMD AOC" for depth/fbbpp 24/32
[  2079.464] (==) AMDGPU(0): Depth 24, (--) framebuffer bpp 32
[  2079.464] (II) AMDGPU(0): Pixel depth = 24 bits stored in 4 bytes (32 bpp pixmaps)
[  2079.464] (==) AMDGPU(0): Default visual is TrueColor
[  2079.464] (==) AMDGPU(0): RGB weight 888
[  2079.464] (II) AMDGPU(0): Using 8 bits per RGB (8 bit DAC)
[  2079.464] (--) AMDGPU(0): Chipset: "Radeon RX 580 Series" (ChipID = 0x67df)
[  2079.464] (II) Loading sub module "fb"
[  2079.464] (II) LoadModule: "fb"
[  2079.464] (II) Loading /usr/lib/xorg/modules/libfb.so
[  2079.464] (II) Module fb: vendor="X.Org Foundation"
[  2079.464] 	compiled for 1.20.11, module version = 1.0.0
[  2079.464] 	ABI class: X.Org ANSI C Emulation, version 0.4
[  2079.464] (II) Loading sub module "dri2"
[  2079.464] (II) LoadModule: "dri2"
[  2079.464] (II) Module "dri2" already built-in
[  2079.494] (II) Loading sub module "glamoregl"
[  2079.494] (II) LoadModule: "glamoregl"
[  2079.494] (II) Loading /usr/lib/xorg/modules/libglamoregl.so
[  2079.496] (II) Module glamoregl: vendor="X.Org Foundation"
[  2079.496] 	compiled for 1.20.11, module version = 1.0.1
[  2079.496] 	ABI class: X.Org ANSI C Emulation, version 0.4
[  2079.502] (II) AMDGPU(0): glamor X acceleration enabled on Radeon RX 580 Series (POLARIS10, DRM 3.40.0, 5.10.0-20-amd64, LLVM 11.0.1)
[  2079.502] (II) AMDGPU(0): glamor detected, initialising EGL layer.
[  2079.502] (==) AMDGPU(0): TearFree property default: auto
[  2079.502] (==) AMDGPU(0): VariableRefresh: disabled
[  2079.502] (II) AMDGPU(0): KMS Pageflipping: enabled
[  2079.502] (II) AMDGPU(0): Output DisplayPort-0 using monitor section AOC
[  2079.502] (II) AMDGPU(0): Output DisplayPort-1 has no monitor section
[  2079.502] (II) AMDGPU(0): Output DisplayPort-2 has no monitor section
[  2079.502] (II) AMDGPU(0): Output HDMI-A-0 has no monitor section
[  2079.503] (II) AMDGPU(0): Output DVI-D-0 has no monitor section
[  2079.503] (II) AMDGPU(0): EDID for output DisplayPort-0
[  2079.503] (II) AMDGPU(0): Manufacturer: AOC  Model: 2402  Serial#: 139027
[  2079.503] (II) AMDGPU(0): Year: 2020  Week: 27
[  2079.503] (II) AMDGPU(0): EDID Version: 1.4
[  2079.503] (II) AMDGPU(0): Digital Display Input
[  2079.503] (II) AMDGPU(0): 8 bits per channel
[  2079.503] (II) AMDGPU(0): Digital interface is DisplayPort
[  2079.503] (II) AMDGPU(0): Max Image Size [cm]: horiz.: 53  vert.: 30
[  2079.503] (II) AMDGPU(0): Gamma: 2.20
[  2079.503] (II) AMDGPU(0): DPMS capabilities: Off
[  2079.503] (II) AMDGPU(0): Supported color encodings: RGB 4:4:4 YCrCb 4:4:4 YCrCb 4:2:2
[  2079.503] (II) AMDGPU(0): First detailed timing is preferred mode
[  2079.503] (II) AMDGPU(0): Preferred mode is native pixel format and refresh rate
[  2079.503] (II) AMDGPU(0): Display is continuous-frequency
[  2079.503] (II) AMDGPU(0): redX: 0.664 redY: 0.309   greenX: 0.275 greenY: 0.670
[  2079.503] (II) AMDGPU(0): blueX: 0.154 blueY: 0.051   whiteX: 0.313 whiteY: 0.329
[  2079.503] (II) AMDGPU(0): Supported established timings:
[  2079.503] (II) AMDGPU(0): 720x400@70Hz
[  2079.503] (II) AMDGPU(0): 640x480@60Hz
[  2079.503] (II) AMDGPU(0): 640x480@67Hz
[  2079.503] (II) AMDGPU(0): 640x480@72Hz
[  2079.503] (II) AMDGPU(0): 640x480@75Hz
[  2079.503] (II) AMDGPU(0): 800x600@56Hz
[  2079.503] (II) AMDGPU(0): 800x600@60Hz
[  2079.503] (II) AMDGPU(0): 800x600@72Hz
[  2079.503] (II) AMDGPU(0): 800x600@75Hz
[  2079.503] (II) AMDGPU(0): 832x624@75Hz
[  2079.503] (II) AMDGPU(0): 1024x768@60Hz
[  2079.503] (II) AMDGPU(0): 1024x768@70Hz
[  2079.503] (II) AMDGPU(0): 1024x768@75Hz
[  2079.503] (II) AMDGPU(0): 1280x1024@75Hz
[  2079.503] (II) AMDGPU(0): Manufacturer's mask: 0
[  2079.503] (II) AMDGPU(0): Supported standard timings:
[  2079.503] (II) AMDGPU(0): #0: hsize: 1920  vsize 1080  refresh: 60  vid: 49361
[  2079.503] (II) AMDGPU(0): #1: hsize: 1280  vsize 1024  refresh: 60  vid: 32897
[  2079.503] (II) AMDGPU(0): #2: hsize: 640  vsize 480  refresh: 100  vid: 26673
[  2079.503] (II) AMDGPU(0): #3: hsize: 640  vsize 480  refresh: 120  vid: 31793
[  2079.503] (II) AMDGPU(0): #4: hsize: 800  vsize 600  refresh: 100  vid: 26693
[  2079.503] (II) AMDGPU(0): #5: hsize: 800  vsize 600  refresh: 120  vid: 31813
[  2079.503] (II) AMDGPU(0): #6: hsize: 1024  vsize 768  refresh: 100  vid: 26721
[  2079.503] (II) AMDGPU(0): #7: hsize: 1024  vsize 768  refresh: 120  vid: 31841
[  2079.503] (II) AMDGPU(0): Supported detailed timing:
[  2079.503] (II) AMDGPU(0): clock: 148.5 MHz   Image Size:  527 x 296 mm
[  2079.503] (II) AMDGPU(0): h_active: 1920  h_sync: 2008  h_sync_end 2052 h_blank_end 2200 h_border: 0
[  2079.503] (II) AMDGPU(0): v_active: 1080  v_sync: 1084  v_sync_end 1089 v_blanking: 1125 v_border: 0
[  2079.503] (II) AMDGPU(0): Serial No: ATNL61A139027
[  2079.503] (II) AMDGPU(0): Monitor name: 24G2W1G4
[  2079.503] (II) AMDGPU(0): Ranges: V min: 48 V max: 144 Hz, H min: 160 H max: 160 kHz, PixClock max 335 MHz
[  2079.503] (II) AMDGPU(0): Supported detailed timing:
[  2079.503] (II) AMDGPU(0): clock: 325.7 MHz   Image Size:  527 x 296 mm
[  2079.503] (II) AMDGPU(0): h_active: 1920  h_sync: 1944  h_sync_end 1976 h_blank_end 2056 h_border: 0
[  2079.503] (II) AMDGPU(0): v_active: 1080  v_sync: 1083  v_sync_end 1088 v_blanking: 1100 v_border: 0
[  2079.503] (II) AMDGPU(0): Supported detailed timing:
[  2079.503] (II) AMDGPU(0): clock: 285.5 MHz   Image Size:  527 x 296 mm
[  2079.503] (II) AMDGPU(0): h_active: 1920  h_sync: 1968  h_sync_end 2000 h_blank_end 2080 h_border: 0
[  2079.503] (II) AMDGPU(0): v_active: 1080  v_sync: 1083  v_sync_end 1088 v_blanking: 1144 v_border: 0
[  2079.503] (II) AMDGPU(0): Supported detailed timing:
[  2079.503] (II) AMDGPU(0): clock: 235.5 MHz   Image Size:  527 x 296 mm
[  2079.503] (II) AMDGPU(0): h_active: 1920  h_sync: 1968  h_sync_end 2000 h_blank_end 2080 h_border: 0
[  2079.503] (II) AMDGPU(0): v_active: 1080  v_sync: 1083  v_sync_end 1088 v_blanking: 1133 v_border: 0
[  2079.503] (II) AMDGPU(0): Supported detailed timing:
[  2079.503] (II) AMDGPU(0): clock: 174.5 MHz   Image Size:  527 x 296 mm
[  2079.503] (II) AMDGPU(0): h_active: 1920  h_sync: 1968  h_sync_end 2000 h_blank_end 2080 h_border: 0
[  2079.503] (II) AMDGPU(0): v_active: 1080  v_sync: 1083  v_sync_end 1088 v_blanking: 1119 v_border: 0
[  2079.503] (II) AMDGPU(0): Number of EDID sections to follow: 1
[  2079.503] (II) AMDGPU(0): EDID (in hex):
[  2079.503] (II) AMDGPU(0): 	00ffffffffffff0005e30224131f0200
[  2079.503] (II) AMDGPU(0): 	1b1e0104a5351e783b0a85aa4f46ab27
[  2079.503] (II) AMDGPU(0): 	0d5054bfef00d1c081803168317c4568
[  2079.503] (II) AMDGPU(0): 	457c6168617c023a801871382d40582c
[  2079.503] (II) AMDGPU(0): 	45000f282100001e000000ff0041544e
[  2079.503] (II) AMDGPU(0): 	4c363141313339303237000000fc0032
[  2079.503] (II) AMDGPU(0): 	344732573147340a20202020000000fd
[  2079.503] (II) AMDGPU(0): 	003090a0a021010a2020202020200169
[  2079.503] (II) AMDGPU(0): 	020327f14c0103051404131f12021190
[  2079.503] (II) AMDGPU(0): 	3f23090707830100006d1a0000020130
[  2079.503] (II) AMDGPU(0): 	90000000000000377f80887038144018
[  2079.503] (II) AMDGPU(0): 	2035000f282100001e866f80a0703840
[  2079.503] (II) AMDGPU(0): 	40302035000f282100001efe5b80a070
[  2079.504] (II) AMDGPU(0): 	383540302035000f282100001e2a4480
[  2079.504] (II) AMDGPU(0): 	a070382740302035000f282100001a00
[  2079.504] (II) AMDGPU(0): 	0000000000000000000000000000004e
[  2079.504] (II) AMDGPU(0): Not using mode "1920x1080" (hsync out of range)
[  2079.504] (II) AMDGPU(0): Not using mode "1920x1080" (hsync out of range)
[  2079.504] (II) AMDGPU(0): Not using mode "1920x1080" (vrefresh out of range)
[  2079.504] (II) AMDGPU(0): Not using mode "1920x1080" (vrefresh out of range)
[  2079.504] (II) AMDGPU(0): Not using mode "1920x1080" (vrefresh out of range)
[  2079.504] (II) AMDGPU(0): Not using mode "1920x1080" (vrefresh out of range)
[  2079.504] (II) AMDGPU(0): Not using mode "1920x1080" (vrefresh out of range)
[  2079.504] (II) AMDGPU(0): Not using mode "1920x1080" (vrefresh out of range)
[  2079.504] (II) AMDGPU(0): Not using mode "1920x1080" (hsync out of range)
[  2079.504] (II) AMDGPU(0): Not using mode "1920x1080" (vrefresh out of range)
[  2079.504] (II) AMDGPU(0): Not using mode "1680x1050" (hsync out of range)
[  2079.504] (II) AMDGPU(0): Not using mode "1280x1024" (vrefresh out of range)
[  2079.504] (II) AMDGPU(0): Not using mode "1280x1024" (hsync out of range)
[  2079.504] (II) AMDGPU(0): Not using mode "1440x900" (hsync out of range)
[  2079.504] (II) AMDGPU(0): Not using mode "1280x800" (hsync out of range)
[  2079.504] (II) AMDGPU(0): Not using mode "1280x720" (hsync out of range)
[  2079.504] (II) AMDGPU(0): Not using mode "1280x720" (hsync out of range)
[  2079.504] (II) AMDGPU(0): Not using mode "1280x720" (vrefresh out of range)
[  2079.504] (II) AMDGPU(0): Not using mode "1024x768" (vrefresh out of range)
[  2079.504] (II) AMDGPU(0): Not using mode "1024x768" (vrefresh out of range)
[  2079.504] (II) AMDGPU(0): Not using mode "1024x768" (vrefresh out of range)
[  2079.504] (II) AMDGPU(0): Not using mode "1024x768" (vrefresh out of range)
[  2079.504] (II) AMDGPU(0): Not using mode "1024x768" (hsync out of range)
[  2079.504] (II) AMDGPU(0): Not using mode "832x624" (vrefresh out of range)
[  2079.504] (II) AMDGPU(0): Not using mode "800x600" (vrefresh out of range)
[  2079.504] (II) AMDGPU(0): Not using mode "800x600" (vrefresh out of range)
[  2079.504] (II) AMDGPU(0): Not using mode "800x600" (vrefresh out of range)
[  2079.504] (II) AMDGPU(0): Not using mode "800x600" (vrefresh out of range)
[  2079.504] (II) AMDGPU(0): Not using mode "800x600" (hsync out of range)
[  2079.504] (II) AMDGPU(0): Not using mode "800x600" (vrefresh out of range)
[  2079.504] (II) AMDGPU(0): Not using mode "720x576" (vrefresh out of range)
[  2079.504] (II) AMDGPU(0): Not using mode "720x480" (hsync out of range)
[  2079.504] (II) AMDGPU(0): Not using mode "720x480" (hsync out of range)
[  2079.504] (II) AMDGPU(0): Not using mode "640x480" (vrefresh out of range)
[  2079.504] (II) AMDGPU(0): Not using mode "640x480" (vrefresh out of range)
[  2079.504] (II) AMDGPU(0): Not using mode "640x480" (vrefresh out of range)
[  2079.504] (II) AMDGPU(0): Not using mode "640x480" (vrefresh out of range)
[  2079.504] (II) AMDGPU(0): Not using mode "640x480" (vrefresh out of range)
[  2079.504] (II) AMDGPU(0): Not using mode "640x480" (hsync out of range)
[  2079.504] (II) AMDGPU(0): Not using mode "640x480" (hsync out of range)
[  2079.504] (II) AMDGPU(0): Not using mode "720x400" (vrefresh out of range)
[  2079.504] (II) AMDGPU(0): No remaining probed modes for output DisplayPort-0
[  2079.504] (II) AMDGPU(0): EDID for output DisplayPort-1
[  2079.504] (II) AMDGPU(0): EDID for output DisplayPort-2
[  2079.504] (II) AMDGPU(0): EDID for output HDMI-A-0
[  2079.504] (II) AMDGPU(0): EDID for output DVI-D-0
[  2079.504] (II) AMDGPU(0): Output DisplayPort-0 connected
[  2079.504] (II) AMDGPU(0): Output DisplayPort-1 disconnected
[  2079.504] (II) AMDGPU(0): Output DisplayPort-2 disconnected
[  2079.504] (II) AMDGPU(0): Output HDMI-A-0 disconnected
[  2079.504] (II) AMDGPU(0): Output DVI-D-0 disconnected
[  2079.504] (WW) AMDGPU(0): Unable to find initial modes
[  2079.504] (EE) AMDGPU(0): Output DisplayPort-0 enabled but has no modes
[  2079.504] (II) AMDGPU(0): mem size init: gart size :78dd1000 vram size: s:feee3000 visible:f2e7000
[  2079.504] (==) AMDGPU(0): DPI set to (96, 96)
[  2079.504] (==) AMDGPU(0): Using gamma correction (1.0, 1.0, 1.0)
[  2079.504] (II) Loading sub module "ramdac"
[  2079.504] (II) LoadModule: "ramdac"
[  2079.504] (II) Module "ramdac" already built-in
[  2079.504] (EE) AMDGPU(0): No modes.
[  2079.504] (II) UnloadModule: "amdgpu"
[  2079.504] (II) UnloadSubModule: "glamoregl"
[  2079.504] (II) Unloading glamoregl
[  2079.504] (II) UnloadSubModule: "fb"
[  2079.504] (II) Unloading fb
[  2079.504] (EE) Screen(s) found, but none have a usable configuration.
[  2079.504] (EE) 
Fatal server error:
[  2079.504] (EE) no screens found(EE) 
[  2079.504] (EE) 
Please consult the The X.Org Foundation support 
	 at http://wiki.x.org
 for help. 
[  2079.504] (EE) Please also check the log file at "/var/log/Xorg.1.log" for additional information.
[  2079.504] (EE) 
[  2079.507] (EE) Server terminated with error (1). Closing log file.

I got it working! I had to change Modeline to a supported one in AOC.conf.

Only issue is my PS/2 keyboard does not seem to react in gaming VM, do I need a USB keyboard for the gaming VM?

Yes it is better, I usually pass a mouse and a keyboard in the VM

For better reactivity in games, I even suggest passthrough-ing a whole USB -controller- to the gaming qube.
Last I used USB -device- passthrough, I found some unacceptable lags for gaming. It was a few years back though, so YMMV !

Of course, your MoBo must have several USB controllers, but it’s usually the case (one on the CPU, one on the chipset). If unsure, some manufacturers have a “block diagram”.
If you don’t have several controllers on the MoBo, the alternative would be a PCIe-to-USB controller which you can then PT to the gaming qube.

Same remarks for audio, I found QEMU rather strange with noises and delay, but again, old tests so YMMV !

(Another remark, sorry for double posting but I forgot and didn’t want to edit, for users using mail).
When I’ve set up, I explicitely excluded the RX580 from the dom0 xorg.conf file, well, rather only included the non-gaming GPU in it like :

# GPU
Section "Device"
	Identifier  "Card1"
	Driver      "nouveau"
	BusID       "PCI:32:0:0"
EndSection

This way I can plug the RX580 in the primary/x16 slot.

PS: I’ve realized too late that you’re using a laptop, but it may be useful to desktop users. I’m not really a laptop user so other solutions may apply though.

I am not using laptop, don’t worry.