Hi
When I attempt VM passthrough of this device to a VM it consistently causes a kernel panic seconds after boot. I have tried it on the 4 different dom0 kernels currently installed on my system and on a couple of VM kernels. I have tried VMs based on Debian 13, Fedora 44, and Ubuntu 24.04.
From searching the forum I see at least one user has done passthrough of an R9700 successfully so I am hopeful I will eventually succeed too. It would be so nice to be able to contribute a positive HCL report at the end of this, as there were a couple other installation obstacles.
Here is my specific hardware:
- Motherboard: Gigabyte B850 AI TOP rev 1.0 (up to date firmware)
- CPU: AMD Ryzen 9950X3D
- GPU: PowerColor Radeon AI PRO R9700 32GB
When the GPU is left attached to dom0 the boot up succeeds. Here are the lines of journalctl -b | grep amdgpu for a successful dom0 boot with the GPU attached:
07:11:45 localhost kernel: amdgpu: vga_switcheroo: detected switching method \_SB_.PCI0.GP17.VGA_.ATPX handle
07:11:45 localhost kernel: amdgpu: ATPX version 1, functions 0x00000000
07:11:45 localhost kernel: amdgpu: Virtual CRAT table created for CPU
07:11:45 localhost kernel: amdgpu: Topology: Add CPU node
07:11:45 localhost kernel: amdgpu 0000:03:00.0: enabling device (0006 -> 0007)
07:11:45 localhost kernel: amdgpu 0000:03:00.0: initializing kernel modesetting (IP DISCOVERY 0x1002:0x7551 0x148C:0x2443 0xC0).
07:11:45 localhost kernel: amdgpu 0000:03:00.0: register mmio base: 0xDFC00000
07:11:45 localhost kernel: amdgpu 0000:03:00.0: register mmio size: 524288
07:11:45 localhost kernel: amdgpu 0000:03:00.0: detected ip block number 0 <common_v1_0_0> (soc24_common)
07:11:45 localhost kernel: amdgpu 0000:03:00.0: detected ip block number 1 <gmc_v12_0_0> (gmc_v12_0)
07:11:45 localhost kernel: amdgpu 0000:03:00.0: detected ip block number 2 <ih_v7_0_0> (ih_v7_0)
07:11:45 localhost kernel: amdgpu 0000:03:00.0: detected ip block number 3 <psp_v14_0_0> (psp)
07:11:45 localhost kernel: amdgpu 0000:03:00.0: detected ip block number 4 <smu_v14_0_0> (smu)
07:11:45 localhost kernel: amdgpu 0000:03:00.0: detected ip block number 5 <dce_v1_0_0> (dm)
07:11:45 localhost kernel: amdgpu 0000:03:00.0: detected ip block number 6 <gfx_v12_0_0> (gfx_v12_0)
07:11:45 localhost kernel: amdgpu 0000:03:00.0: detected ip block number 7 <sdma_v7_0_0> (sdma_v7_0)
07:11:45 localhost kernel: amdgpu 0000:03:00.0: detected ip block number 8 <vcn_v5_0_0> (vcn_v5_0_0)
07:11:45 localhost kernel: amdgpu 0000:03:00.0: detected ip block number 9 <jpeg_v5_0_0> (jpeg_v5_0_0)
07:11:45 localhost kernel: amdgpu 0000:03:00.0: detected ip block number 10 <mes_v12_0_0> (mes_v12_0)
07:11:45 localhost kernel: amdgpu 0000:03:00.0: Fetched VBIOS from VFCT
07:11:45 localhost kernel: amdgpu 0000:03:00.0: [drm] ATOM BIOS: 113-EXT119250-100
07:11:45 localhost kernel: amdgpu 0000:03:00.0: Trusted Memory Zone (TMZ) feature not supported
07:11:45 localhost kernel: amdgpu 0000:03:00.0: MEM ECC is active.
07:11:45 localhost kernel: amdgpu 0000:03:00.0: SRAM ECC is not presented.
07:11:45 localhost kernel: amdgpu 0000:03:00.0: RAS INFO: ras initialized successfully, hardware ability[101] ras_mask[101]
07:11:45 localhost kernel: amdgpu 0000:03:00.0: vm size is 262144 GB, 4 levels, block size is 9-bit, fragment size is 9-bit
07:11:45 localhost kernel: amdgpu 0000:03:00.0: VRAM: 32624M 0x0000008000000000 - 0x00000087F6FFFFFF (32624M used)
07:11:45 localhost kernel: amdgpu 0000:03:00.0: GART: 512M 0x0000000000000000 - 0x000000001FFFFFFF
07:11:45 localhost kernel: amdgpu 0000:03:00.0: [drm] Detected VRAM RAM=32624M, BAR=32768M
07:11:45 localhost kernel: amdgpu 0000:03:00.0: [drm] RAM width 256bits GDDR6
07:11:47 localhost kernel: amdgpu 0000:03:00.0: 32624M of VRAM memory ready
07:11:47 localhost kernel: amdgpu 0000:03:00.0: 1942M of GTT memory ready.
07:11:47 localhost kernel: amdgpu 0000:03:00.0: [drm] GART: num cpu pages 131072, num gpu pages 131072
07:11:47 localhost kernel: amdgpu 0000:03:00.0: [drm] PCIE GART of 512M enabled (table at 0x00000087D6B00000).
07:11:47 localhost kernel: amdgpu 0000:03:00.0: [drm] Loading DMUB firmware via PSP: version=0x0A003A00
07:11:47 localhost kernel: amdgpu 0000:03:00.0: [VCN instance 0] Found VCN firmware Version ENC: 1.12 DEC: 9 VEP: 0 Revision: 26
07:11:47 localhost kernel: amdgpu 0000:03:00.0: MES: vmid_mask_mmhub 0x0000ff00, vmid_mask_gfxhub 0x0000ff00
07:11:47 localhost kernel: amdgpu 0000:03:00.0: MES: gfx_hqd_mask 0x000000fe, compute_hqd_mask 0x0000000c, sdma_hqd_mask 0x000000fc
07:11:47 localhost kernel: amdgpu 0000:03:00.0: GECC is disabled, set amdgpu_ras_enable=1 to enable GECC in next boot cycle if needed
07:11:47 localhost kernel: amdgpu 0000:03:00.0: RAP: optional rap ta ucode is not available
07:11:47 localhost kernel: amdgpu 0000:03:00.0: SECUREDISPLAY: optional securedisplay ta ucode is not available
07:11:47 localhost kernel: amdgpu 0000:03:00.0: smu driver if version = 0x0000002e, smu fw if version = 0x00000033, smu fw program = 0, smu fw version = 0x00684c00 (104.76.0)
07:11:47 localhost kernel: amdgpu 0000:03:00.0: SMU is initialized successfully!
07:11:47 localhost kernel: amdgpu 0000:03:00.0: [drm] Display Core v3.2.369 initialized on DCN 4.0.1
07:11:47 localhost kernel: amdgpu 0000:03:00.0: [drm] DP-HDMI FRL PCON supported
07:11:47 localhost kernel: amdgpu 0000:03:00.0: [drm] DMUB hardware initialized: version=0x0A003A00
07:11:47 localhost kernel: amdgpu 0000:03:00.0: [drm] DP-1: PSR support 0, DC PSR ver -1, sink PSR ver 0 DPCD caps 0x0 su_y_granularity 0
07:11:47 localhost kernel: amdgpu 0000:03:00.0: [drm] DP-2: PSR support 0, DC PSR ver -1, sink PSR ver 0 DPCD caps 0x0 su_y_granularity 0
07:11:47 localhost kernel: amdgpu 0000:03:00.0: [drm] DP-3: PSR support 0, DC PSR ver -1, sink PSR ver 0 DPCD caps 0x0 su_y_granularity 0
07:11:47 localhost kernel: amdgpu 0000:03:00.0: [drm] DP-4: PSR support 0, DC PSR ver -1, sink PSR ver 0 DPCD caps 0x0 su_y_granularity 0
07:11:47 localhost kernel: amdgpu 0000:03:00.0: program CP_MES_CNTL : 0x4000000
07:11:47 localhost kernel: amdgpu 0000:03:00.0: program CP_MES_CNTL : 0xc000000
07:11:47 localhost kernel: amdgpu: Virtual CRAT table created for GPU
07:11:47 localhost kernel: amdgpu: Topology: Add GPU node [0x1002:0x7551]
07:11:47 localhost kernel: amdgpu 0000:03:00.0: SE 4, SH per SE 2, CU per SH 8, active_cu_number 64
07:11:47 localhost kernel: amdgpu 0000:03:00.0: ring gfx_0.0.0 uses VM inv eng 0 on hub 0
07:11:47 localhost kernel: amdgpu 0000:03:00.0: ring comp_1.0.0 uses VM inv eng 1 on hub 0
07:11:47 localhost kernel: amdgpu 0000:03:00.0: ring comp_1.1.0 uses VM inv eng 4 on hub 0
07:11:47 localhost kernel: amdgpu 0000:03:00.0: ring comp_1.0.1 uses VM inv eng 7 on hub 0
07:11:47 localhost kernel: amdgpu 0000:03:00.0: ring comp_1.1.1 uses VM inv eng 8 on hub 0
07:11:47 localhost kernel: amdgpu 0000:03:00.0: ring sdma0 uses VM inv eng 9 on hub 0
07:11:47 localhost kernel: amdgpu 0000:03:00.0: ring sdma1 uses VM inv eng 10 on hub 0
07:11:47 localhost kernel: amdgpu 0000:03:00.0: ring vcn_unified_0 uses VM inv eng 0 on hub 8
07:11:47 localhost kernel: amdgpu 0000:03:00.0: ring jpeg_dec uses VM inv eng 1 on hub 8
07:11:47 localhost kernel: amdgpu: HMM registered 32624MB device memory
07:11:47 localhost kernel: amdgpu 0000:03:00.0: Using BACO for runtime pm
07:11:47 localhost kernel: amdgpu 0000:03:00.0: [drm] Registered 4 planes with drm panic
07:11:47 localhost kernel: [drm] Initialized amdgpu 3.64.0 for 0000:03:00.0 on minor 0
07:11:47 localhost kernel: amdgpu 0000:03:00.0: [drm] Cannot find any crtc or sizes
07:11:47 localhost kernel: amdgpu 0000:12:00.0: enabling device (0006 -> 0007)
07:11:47 localhost kernel: amdgpu 0000:12:00.0: initializing kernel modesetting (IP DISCOVERY 0x1002:0x13C0 0x1458:0xD000 0xC9).
07:11:47 localhost kernel: amdgpu 0000:12:00.0: register mmio base: 0xDFB00000
07:11:47 localhost kernel: amdgpu 0000:12:00.0: register mmio size: 524288
07:11:47 localhost kernel: amdgpu 0000:12:00.0: detected ip block number 0 <common_v1_0_0> (nv_common)
07:11:47 localhost kernel: amdgpu 0000:12:00.0: detected ip block number 1 <gmc_v10_0_0> (gmc_v10_0)
07:11:47 localhost kernel: amdgpu 0000:12:00.0: detected ip block number 2 <ih_v5_0_0> (navi10_ih)
07:11:47 localhost kernel: amdgpu 0000:12:00.0: detected ip block number 3 <psp_v13_0_0> (psp)
07:11:47 localhost kernel: amdgpu 0000:12:00.0: detected ip block number 4 <smu_v13_0_0> (smu)
07:11:47 localhost kernel: amdgpu 0000:12:00.0: detected ip block number 5 <dce_v1_0_0> (dm)
07:11:47 localhost kernel: amdgpu 0000:12:00.0: detected ip block number 6 <gfx_v10_0_0> (gfx_v10_0)
07:11:47 localhost kernel: amdgpu 0000:12:00.0: detected ip block number 7 <sdma_v5_2_0> (sdma_v5_2)
07:11:47 localhost kernel: amdgpu 0000:12:00.0: detected ip block number 8 <vcn_v3_0_0> (vcn_v3_0)
07:11:47 localhost kernel: amdgpu 0000:12:00.0: detected ip block number 9 <jpeg_v3_0_0> (jpeg_v3_0)
07:11:47 localhost kernel: amdgpu 0000:12:00.0: Fetched VBIOS from VFCT
07:11:47 localhost kernel: amdgpu 0000:12:00.0: [drm] ATOM BIOS: 102-RAPHAEL-008
07:11:47 localhost kernel: amdgpu 0000:12:00.0: vgaarb: deactivate vga console
07:11:47 localhost kernel: amdgpu 0000:12:00.0: Trusted Memory Zone (TMZ) feature disabled as experimental (default)
07:11:47 localhost kernel: amdgpu 0000:12:00.0: vm size is 262144 GB, 4 levels, block size is 9-bit, fragment size is 9-bit
07:11:47 localhost kernel: amdgpu 0000:12:00.0: VRAM: 2048M 0x000000F400000000 - 0x000000F47FFFFFFF (2048M used)
07:11:47 localhost kernel: amdgpu 0000:12:00.0: GART: 1024M 0x0000000000000000 - 0x000000003FFFFFFF
07:11:47 localhost kernel: amdgpu 0000:12:00.0: [drm] Detected VRAM RAM=2048M, BAR=2048M
07:11:47 localhost kernel: amdgpu 0000:12:00.0: [drm] RAM width 128bits DDR5
07:11:47 localhost kernel: amdgpu 0000:12:00.0: 2048M of VRAM memory ready
07:11:47 localhost kernel: amdgpu 0000:12:00.0: 1942M of GTT memory ready.
07:11:47 localhost kernel: amdgpu 0000:12:00.0: [drm] GART: num cpu pages 262144, num gpu pages 262144
07:11:47 localhost kernel: amdgpu 0000:12:00.0: [drm] PCIE GART of 1024M enabled (table at 0x000000F47FC00000).
07:11:47 localhost kernel: amdgpu 0000:12:00.0: [drm] Loading DMUB firmware via PSP: version=0x05003000
07:11:47 localhost kernel: amdgpu 0000:12:00.0: [drm] use_doorbell being set to: [true]
07:11:47 localhost kernel: amdgpu 0000:12:00.0: [VCN instance 0] Found VCN firmware Version ENC: 1.33 DEC: 4 VEP: 0 Revision: 15
07:11:47 localhost kernel: amdgpu 0000:12:00.0: reserve 0xa00000 from 0xf47e000000 for PSP TMR
07:11:47 localhost kernel: amdgpu 0000:12:00.0: RAS: optional ras ta ucode is not available
07:11:48 localhost kernel: amdgpu 0000:12:00.0: RAP: optional rap ta ucode is not available
07:11:48 localhost kernel: amdgpu 0000:12:00.0: SECUREDISPLAY: optional securedisplay ta ucode is not available
07:11:48 localhost kernel: amdgpu 0000:12:00.0: SMU is initialized successfully!
07:11:48 localhost kernel: amdgpu 0000:12:00.0: [drm] Display Core v3.2.369 initialized on DCN 3.1.5
07:11:48 localhost kernel: amdgpu 0000:12:00.0: [drm] DP-HDMI FRL PCON supported
07:11:48 localhost kernel: amdgpu 0000:12:00.0: [drm] DMUB hardware initialized: version=0x05003000
07:11:48 localhost kernel: amdgpu 0000:12:00.0: [drm] HDMI-A-1: PSR support 0, DC PSR ver -1, sink PSR ver 0 DPCD caps 0x0 su_y_granularity 0
07:11:48 localhost kernel: amdgpu 0000:12:00.0: [drm] DP-5: PSR support 0, DC PSR ver -1, sink PSR ver 0 DPCD caps 0x0 su_y_granularity 0
07:11:48 localhost kernel: amdgpu 0000:12:00.0: [drm] DP-6: PSR support 0, DC PSR ver -1, sink PSR ver 0 DPCD caps 0x0 su_y_granularity 0
07:11:48 localhost kernel: amdgpu 0000:12:00.0: [drm] DP-7: PSR support 0, DC PSR ver -1, sink PSR ver 0 DPCD caps 0x0 su_y_granularity 0
07:11:48 localhost kernel: amdgpu 0000:12:00.0: kiq ring mec 2 pipe 1 q 0
07:11:48 localhost kernel: amdgpu: Virtual CRAT table created for GPU
07:11:48 localhost kernel: amdgpu: Topology: Add GPU node [0x1002:0x13c0]
07:11:48 localhost kernel: amdgpu 0000:12:00.0: SE 1, SH per SE 1, CU per SH 2, active_cu_number 2
07:11:48 localhost kernel: amdgpu 0000:12:00.0: ring gfx_0.0.0 uses VM inv eng 0 on hub 0
07:11:48 localhost kernel: amdgpu 0000:12:00.0: ring gfx_0.1.0 uses VM inv eng 1 on hub 0
07:11:48 localhost kernel: amdgpu 0000:12:00.0: ring comp_1.0.0 uses VM inv eng 4 on hub 0
07:11:48 localhost kernel: amdgpu 0000:12:00.0: ring comp_1.1.0 uses VM inv eng 5 on hub 0
07:11:48 localhost kernel: amdgpu 0000:12:00.0: ring comp_1.2.0 uses VM inv eng 6 on hub 0
07:11:48 localhost kernel: amdgpu 0000:12:00.0: ring comp_1.3.0 uses VM inv eng 7 on hub 0
07:11:48 localhost kernel: amdgpu 0000:12:00.0: ring comp_1.0.1 uses VM inv eng 8 on hub 0
07:11:48 localhost kernel: amdgpu 0000:12:00.0: ring comp_1.1.1 uses VM inv eng 9 on hub 0
07:11:48 localhost kernel: amdgpu 0000:12:00.0: ring comp_1.2.1 uses VM inv eng 10 on hub 0
07:11:48 localhost kernel: amdgpu 0000:12:00.0: ring comp_1.3.1 uses VM inv eng 11 on hub 0
07:11:48 localhost kernel: amdgpu 0000:12:00.0: ring kiq_0.2.1.0 uses VM inv eng 12 on hub 0
07:11:48 localhost kernel: amdgpu 0000:12:00.0: ring sdma0 uses VM inv eng 13 on hub 0
07:11:48 localhost kernel: amdgpu 0000:12:00.0: ring vcn_dec_0 uses VM inv eng 0 on hub 8
07:11:48 localhost kernel: amdgpu 0000:12:00.0: ring vcn_enc_0.0 uses VM inv eng 1 on hub 8
07:11:48 localhost kernel: amdgpu 0000:12:00.0: ring vcn_enc_0.1 uses VM inv eng 4 on hub 8
07:11:48 localhost kernel: amdgpu 0000:12:00.0: ring jpeg_dec uses VM inv eng 5 on hub 8
07:11:48 localhost kernel: amdgpu: HMM registered 2048MB device memory
07:11:48 localhost kernel: amdgpu 0000:12:00.0: Runtime PM not available
07:11:48 localhost kernel: amdgpu 0000:12:00.0: [drm] Registered 4 planes with drm panic
07:11:48 localhost kernel: [drm] Initialized amdgpu 3.64.0 for 0000:12:00.0 on minor 1
07:11:48 localhost kernel: fbcon: amdgpudrmfb (fb0) is primary device
07:11:48 localhost kernel: amdgpu 0000:12:00.0: [drm] fb0: amdgpudrmfb frame buffer device
07:12:04 dom0 kernel: amdgpu 0000:03:00.0: [drm] PCIE GART of 512M enabled (table at 0x00000087D6B00000).
07:12:04 dom0 kernel: amdgpu 0000:03:00.0: PSP is resuming...
07:12:04 dom0 kernel: amdgpu 0000:03:00.0: GECC is disabled, set amdgpu_ras_enable=1 to enable GECC in next boot cycle if needed
07:12:04 dom0 kernel: amdgpu 0000:03:00.0: RAP: optional rap ta ucode is not available
07:12:04 dom0 kernel: amdgpu 0000:03:00.0: SECUREDISPLAY: optional securedisplay ta ucode is not available
07:12:04 dom0 kernel: amdgpu 0000:03:00.0: SMU is resuming...
07:12:04 dom0 kernel: amdgpu 0000:03:00.0: smu driver if version = 0x0000002e, smu fw if version = 0x00000033, smu fw program = 0, smu fw version = 0x00684c00 (104.76.0)
07:12:04 dom0 kernel: amdgpu 0000:03:00.0: SMU is resumed successfully!
07:12:04 dom0 kernel: amdgpu 0000:03:00.0: program CP_MES_CNTL : 0x4000000
07:12:04 dom0 kernel: amdgpu 0000:03:00.0: program CP_MES_CNTL : 0xc000000
07:12:04 dom0 kernel: amdgpu 0000:03:00.0: [drm] DMUB hardware initialized: version=0x0A003A00
07:12:04 dom0 kernel: amdgpu 0000:03:00.0: [drm] Cannot find any crtc or sizes
07:12:04 dom0 kernel: amdgpu 0000:03:00.0: ring gfx_0.0.0 uses VM inv eng 0 on hub 0
07:12:04 dom0 kernel: amdgpu 0000:03:00.0: ring comp_1.0.0 uses VM inv eng 1 on hub 0
07:12:04 dom0 kernel: amdgpu 0000:03:00.0: ring comp_1.1.0 uses VM inv eng 4 on hub 0
07:12:04 dom0 kernel: amdgpu 0000:03:00.0: ring comp_1.0.1 uses VM inv eng 7 on hub 0
07:12:04 dom0 kernel: amdgpu 0000:03:00.0: ring comp_1.1.1 uses VM inv eng 8 on hub 0
07:12:04 dom0 kernel: amdgpu 0000:03:00.0: ring sdma0 uses VM inv eng 9 on hub 0
07:12:04 dom0 kernel: amdgpu 0000:03:00.0: ring sdma1 uses VM inv eng 10 on hub 0
07:12:04 dom0 kernel: amdgpu 0000:03:00.0: ring vcn_unified_0 uses VM inv eng 0 on hub 8
07:12:04 dom0 kernel: amdgpu 0000:03:00.0: ring jpeg_dec uses VM inv eng 1 on hub 8
07:12:04 dom0 kernel: amdgpu 0000:03:00.0: [drm] Cannot find any crtc or sizes
07:12:04 dom0 kernel: snd_hda_intel 0000:12:00.1: bound 0000:12:00.0 (ops amdgpu_dm_audio_component_bind_ops [amdgpu])
07:12:04 dom0 kernel: snd_hda_intel 0000:03:00.1: bound 0000:03:00.0 (ops amdgpu_dm_audio_component_bind_ops [amdgpu])
07:12:08 dom0 kernel: amdgpu 0000:03:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=none:owns=none
07:12:08 dom0 kernel: amdgpu 0000:12:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=none:owns=none
07:12:17 dom0 kernel: amdgpu 0000:03:00.0: [drm] PCIE GART of 512M enabled (table at 0x00000087D6B00000).
07:12:17 dom0 kernel: amdgpu 0000:03:00.0: PSP is resuming...
07:12:17 dom0 kernel: amdgpu 0000:03:00.0: GECC is disabled, set amdgpu_ras_enable=1 to enable GECC in next boot cycle if needed
07:12:17 dom0 kernel: amdgpu 0000:03:00.0: RAP: optional rap ta ucode is not available
07:12:17 dom0 kernel: amdgpu 0000:03:00.0: SECUREDISPLAY: optional securedisplay ta ucode is not available
07:12:17 dom0 kernel: amdgpu 0000:03:00.0: SMU is resuming...
07:12:17 dom0 kernel: amdgpu 0000:03:00.0: smu driver if version = 0x0000002e, smu fw if version = 0x00000033, smu fw program = 0, smu fw version = 0x00684c00 (104.76.0)
07:12:17 dom0 kernel: amdgpu 0000:03:00.0: SMU is resumed successfully!
07:12:17 dom0 kernel: amdgpu 0000:03:00.0: program CP_MES_CNTL : 0x4000000
07:12:17 dom0 kernel: amdgpu 0000:03:00.0: program CP_MES_CNTL : 0xc000000
07:12:17 dom0 kernel: amdgpu 0000:03:00.0: [drm] DMUB hardware initialized: version=0x0A003A00
07:12:17 dom0 kernel: amdgpu 0000:03:00.0: [drm] Cannot find any crtc or sizes
07:12:17 dom0 kernel: amdgpu 0000:03:00.0: ring gfx_0.0.0 uses VM inv eng 0 on hub 0
07:12:17 dom0 kernel: amdgpu 0000:03:00.0: ring comp_1.0.0 uses VM inv eng 1 on hub 0
07:12:17 dom0 kernel: amdgpu 0000:03:00.0: ring comp_1.1.0 uses VM inv eng 4 on hub 0
07:12:17 dom0 kernel: amdgpu 0000:03:00.0: ring comp_1.0.1 uses VM inv eng 7 on hub 0
07:12:17 dom0 kernel: amdgpu 0000:03:00.0: ring comp_1.1.1 uses VM inv eng 8 on hub 0
07:12:17 dom0 kernel: amdgpu 0000:03:00.0: ring sdma0 uses VM inv eng 9 on hub 0
07:12:17 dom0 kernel: amdgpu 0000:03:00.0: ring sdma1 uses VM inv eng 10 on hub 0
07:12:17 dom0 kernel: amdgpu 0000:03:00.0: ring vcn_unified_0 uses VM inv eng 0 on hub 8
07:12:17 dom0 kernel: amdgpu 0000:03:00.0: ring jpeg_dec uses VM inv eng 1 on hub 8
07:12:17 dom0 kernel: amdgpu 0000:03:00.0: [drm] Cannot find any crtc or sizes
When the GPU is passed through to a VM that soon kernel panics, here is an example of the relevant lines of /var/log/xen/console/guest-gpu-passthrough-vm.log, cleaned up by hand:
((oops too long, will post in a follow up comment))
Perhaps the first relevant difference is this line:
amdgpu 0000:00:06.0: System can't access extended configuration space, please check!!
And the panic itself is preceded by this set of lines:
amdgpu 0000:00:06.0: MES(1) failed to respond to msg=SET_HW_RSRC
amdgpu 0000:00:06.0: hw_init of IP block <gfx_v12_0> failed -110
amdgpu 0000:00:06.0: amdgpu_device_ip_init failed
amdgpu 0000:00:06.0: Fatal error during GPU init
But this is surmise.
Some other notes. Here I believe are the VGA and Audio devices of the card, as shown by lspci in dom0:
03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 [Radeon AI PRO R9700] [1002:7551] (rev c0) (prog-if 00 [VGA controller])
Subsystem: Tul Corporation / PowerColor Device [148c:2443]
Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Interrupt: pin A routed to IRQ 24
Region 0: Memory at f000000000 (64-bit, prefetchable) [size=32G]
Region 2: Memory at f800000000 (64-bit, prefetchable) [size=256M]
Region 4: I/O ports at f000 [size=256]
Region 5: Memory at dfc00000 (32-bit, non-prefetchable) [size=512K]
Expansion ROM at dfc80000 [disabled] [size=128K]
Capabilities: <access denied>
Kernel driver in use: pciback
Kernel modules: amdgpu
03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 HDMI/DP Audio Controller [1002:ab40]
Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 HDMI/DP Audio Controller [1002:ab40]
Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Interrupt: pin B routed to IRQ 25
Region 0: Memory at dfca0000 (32-bit, non-prefetchable) [size=16K]
Capabilities: <access denied>
Kernel driver in use: pciback
Kernel modules: snd_hda_intel
You can see they are both Kernel driver in use: pciback as I have hidden them from dom0 with the kernel argument rd.qubes.hide_pci=03:00.0,03:00.1.
Both devices are being passed to the VM, which is set to virtualization mode HVM. I have tried passing only the GPU device also.
Could it be I need to pass additional devices? Is there anything suggestive in the boot journals? Any other things I should try?
Thank you for your help. Qubes would seem a good choice for GPU experimentation given its opportunity for security isolation and rollback. I hope I will get this working.