Finalizing an HVM GPU passthrough (could use some advice)

User Support
1

So I’ve come a long way in implementing some GPU passthrough qubes in order to utilize my modern video card. I currently have a (seemingly) fully functional Windows 10 Qube with GPU and USB passthrough working rather splendidly (it feels faster than running virtman/kvm for sure).

I’m actually really excited about this, as I plan to eventually move my day to day computing to my Qubes machine, and this is one big step in that direction.

My Linux Qube, after quite a bit of tinkering, boots up into the CLI (I haven’t installed X yet), but the only problem is that it doesn’t actually have any display associated with it.

I can run “start console in disposible qube” from Qube Manager and it brings me to a login prompt where I can confirm through lsmod and lspci that my amdgpu driver modules are loaded. But it never outputs to any of the monitor ports, I’ve tried all four.

Starting the Linux GPU qube clears the leftover bootup messages from the monitor plugged into the passed through GPU, as if it’s trying to initialize the display. But I’ve been unable to get anything else out of the display.

Searching for how to enumerate or change displays in Linux seems to assume that I’m already running an X server, which I am not. Has anyone else encountered similar issues, or have any ideas on how to further troubleshoot this issue?

Passed through card is a AMD Navi 10 Radeon RX5700, running on a Ryzen 7-3700 with an older Radeon card for Dom0. DomGPU is currently running kernel v5.9 (due to incompatibility of later kernels)

Appreciate any advice!

2

Do you get the boot messages on a separate “display” than the one directly on the GPU ?
I’m not sure I understand fully what you do exactly, in what order, booting two times the same qube ?

Did you try some kernel command line options ?
Stuff like (no)modeset, or forcing a console gfx mode (ie. resolution).
There are options to set the gfx modes in bootloaders. Is the GRUB/Lilo menu showing somewhere ?
Maybe set a huge timer on the bootloader to allow you to test outputs.
Also you can check what the xen console outputs.
(I’ve never used GPU PT without X though, so it’s what it’s worth ^^).

3

Thank you for your reply

When I start the computer, Monitor2 (attached to GPU2) has the normal bios/linux boot messages up to a point where it says “vga_switcheroo”. At that point execution turns over to Monitor1 (attached to GPU1), and GPU2 is then configured to pass through (assigned pciback driver via GRUB). Qubes then starts up just fine on Monitor1.
Monitor2 is never actually cleared - it still displays the last thing that was sent to it (lines of text ending with “vga_switcheroo”).

So far that’s all expected behaviour, as far as I can tell.

Now when I start up my Qube_GPU2_passthrough, with GPU2 attached to it, it clears the leftover boot messages from Monitor2 and sets it to a blank screen, as if it is initializing the monitor. But there is never any other output to that screen.

I can log into a CLI by selecting “start console in qube” from Qube Manager.

So I have Qube_GPU2_passthrough with a running Linux command line in what seems to be a stable system, but it doesn’t output to the any of the monitor ports of the attached GPU2.

I hope that clears it up, I’m not always very good at getting my point across succinctly.

I do have some kernel options active in the GPU2 qube. They include amdgpu.dc=0, pci=nomsi, and a couple others that were suggested in this forum, but nothing like forcing a console resolution like you say. Would it be worth trying to change the resolution via kernel command line if I’m not even getting a display in the first place?

Do individual Qubes allow you to access GRUB? I guess that might depend on my particular HVM?

Oh, and I definitely plan on using X, but I figure that I should probably get my CLI working correctly before I start trying to debug an X server. ATM, if I run my “startx” script from my GPU Qube, I get a “No displays found” error, so it seems my current issue precludes X from starting anyway.

(I don’t mean to sound high maintenance, just trying to be thorough. Appreciate your help)

4

When debugging it’s better to get more info than not enough ; )

I dunno what your kernel options do ? But forcing a resolution (look for “modeset”) can be of help.
Maybe the fact you don’t get any display is because the resolution is not good or badly detected ?
Or it’s because the display is disabled. You on a workstation right ?
That’s why I said to force the most small resolutions (like 640x480), in bootloader and/or kernel cmdline.
I can’t try an HVM on my Qubes install so I don’t know how it behaves, but usually on Xen you can output the bios/bootloader on a VNC console, then the PT GPU gets the output.
One trick with HVM is booting without a hard drive or cdrom, and force the network boot : the BIOS will be in a loop trying to boot the network, and you have time to figure out where this damn output goes ^^ (note: I specifically wrote BIOS, not UEFI).
If you installed and tried startx, you should read the Xorg.log file, maybe there are useful error messages (IIRC, grep for “EE” which means errors).
You could also try looking for dmesg output, and console logs in /var/log/xen/console/* (if you activated them).
PS: the HVM is booting with (Sea)BIOS or UEFI ? I remember a post on this forum where UEFI (but on dom0) was posing a problem with NVRAM or stuff like that.

5

Ok, this is all good information, thank you. I will try some of these things that you suggest with the resolution and such.

I’m not entirely sure what qualifies as a workstation, but it’s a non-server personal computer, nothing special. The Error in my X log was “No screen found”, but I’ll parse through it a little more thoroughly.

I’m not sure about the BIOS/UEFI - I based the qube off of the standard Fedora Template. Honestly it never occurred to me that a VM would even need a BIOS.

This gives me some guidance in my searching, your expertise is much appreciated.

6

I’m not entirely sure what qualifies as a workstation, but it’s a non-server personal computer, nothing special. The Error in my X log was “No screen found”, but I’ll parse through it a little more thoroughly.

Sorry, I meant “desktop” as opposed to laptop, not workstation.
(A workstation is usually a very powerful desktop PC, for crunching
numbers).
“No screen found”, maybe “edid” or “efifb” related stuff ?
dom0 is booting in BIOS/CSM or UEFI mode ?

PS: I don’t know if it’s still useful, but I have a somewhat similar
setup, and I had to tell in dom0 xorg.conf to only use the dom0 GPU (so
by explicitly declaring only the dom0 GPU, and omitting the other one).

I’m not sure about the BIOS/UEFI - I based the qube off of the standard Fedora Template. Honestly it never occurred to me that a VM would even need a BIOS.

The default type in Qubes is PVH (PV enHanced ^^). The other type is HVM.
PV/PVH qubes don’t emulate any real hardware, as the qube kernel is
prepared to run like that, it knows it runs fully virtualized.
HVM qubes have to emulate hardware, to “look like real hardware”, to
support unmodified guest OSes like Windows.

Only HVM qubes need an emulated BIOS/UEFI, to initialize stuff and boot
from somewhere, as on a real PC.
I asked which firmware you’re using because UEFI introduced new problems
when booting GPUs …

This gives me some guidance in my searching, your expertise is much appreciated.

Expertise that’s a big word, those are just pointers, but thanks ! ^^
Provide logs if you’re comfy ! Maybe the qube dmesg/syslog show some error.
Good luck pinpointing your problem !

7

I will need some time to absorb this, and also to sleep :slight_smile: I will post more information and perhaps some logs in the next day or so, after I have had a chance to apply what I’ve gleaned so far. Thank you.

8

Something very interesting happened today when I started my GPU qube in order to pull logs. I went to run my startx script (“sudo startx xfce4 – :1 -config /home/user/AOC.conf” being the operating line, taken from neowutrans gaming HVM guide)
and the second monitor turned white, with a little popup in the corner, only bit I caught was something about 240 some odd packages having updates available.

Now, for some reason I’m unaware of, I can start an X session that seems to work at least a little bit - it throws a ton of errors but it’s obvious that it has some sort of access to the correct screen.
I do find it odd that my CLI never shows up on that screen though, even though I can access it through the Qube Manager.

Isn’t expected behaviour for the command line interface to be accessible through the ports in the GPU? The same ones X11 uses?

Xorg log

9

Some idea:

  • In the AOC.conf, verify that you modified the values for “Modeline” to fit your screen
  • Instead of sudo startx xfce4 – :1 -config /home/user/AOC.conf
    try
    sudo startx /usr/bin/xfce4-session -- :1 -config /home/user/AOC.conf

If I understood this question correctly "Isn’t expected behaviour for the command line interface to be accessible through the ports in the GPU? ", then the answer is no.
Qubes gui try to launch things on Xorg session 0 “:0”, you are explicitly assigning the GPU to xorg session “:1”.
Passthrough a mouse + keyboard and you can interact with “:1”

10

Well if it isn’t the guide-writer themself!

And I gotta tell you my most sincere thanks for the advice. I changed the Modelines in my config and used “xfce4-session” and now I have a passed through XFCE session that looks fantastic. It looks like there might be some stability issues (still getting “fence fallback spam in my kernel log” but I don’t have much time to do a lot with it today. I’ve finally reached a goal I’ve been aiming for for a while though, with all your help. thanks!