MSI X99A GG support?

User Support Hardware Issues
1

Greetings,
I have been attempting (unsuccessfully) to get Xen 4.12-4.15 to work on a CentOS 7 system for the past month and a half. I’ve been mainly having problems with OVMF hangs and VGA Passthrough issues.

I would like to know if you all would think I’d have any better luck trying to get Qubes OS running on this hardware. I know the best way is to try, but after all the hassle and the lack of any kind of support for Xen or OVMF (and I really tried my best to get some help from both xen-users and edk2 mailing lists/IRC, and numerous google searches), I was unable to get it to work.

I understand the cause of some of my VGA passthrough issues, but the OVMF hangs when I try to do any kind of hardware passthrough are a complete mystery. I was able to get debug versions of OVMF working to give me specific error messages why, but there is no documentation I can find which explains what the error messages even mean, let alone what could be causing them, and no one in any user group seems to know anything about them.

Anyway, I know this isn’t a Xen/OVMF help forum, but I was curious if anyone thinks I may have any realistic chance of getting Qubes to work on a system which seems to have so much trouble with Xen/OVMF as it is.

Thank you for your time and consideration!

2

OK, so I followed my own advice, and installed 4.12. It was extremely/painfully slow going through the install GUI, and the partitioning setup was TERRIBLE. I finally gave up and just let it do whatever it wanted automatically. I have no idea what it did.

I finished the install and restarted. Once the GUI came back up, I noticed that it was even slower. The mouse cursor updates at 1Hz, which makes clicking on menus and tiny buttons an exercise in frustration. After a little initial digging around in the menus, I couldn’t find anything related to a hardware control panel (to see what hardware it thought I had). I eventually just opened a terminal window and started checking some things. Xentop reported that dom0 was chewing up over 110% CPU (fun, that), and top reported the culprit as Xorg. Clearly there’s something wrong with the Xorg setup, like it is running 100% software draws (the screen updates are like a really slow blt function).

Just for clarity, here are my hardware specs:

MSI X99A GG
Intel Xeon 2697A v4 Broadwell-EP 16C/32T
128GB DDR3 RAM
512GB NVME SSD I am using as the boot drive.
Sapphire Nitro+ Radeon RX6950XT 16GB
XFX Radeon RX 550 4GB
USB Keyboard/Mouse
Killer E2400 Dual 1G Ethernet
Killer E1535 Wireless network adapter
Realtek ALC1150 codec
ESS ES9018K2M Codec

The RX6950XT is the primary video adapter, and I want the dom0 to use the RX550. I can not change that, as there is nothing in the BIOS that allows selecting “default” or “boot” adapters (boy do I wish there was :-/ ), and I cannot move the adapters around to different slots due to mechanical/thermal limitations in the case. I would like to switch dom0 over to the RX550, and pass through the RX6950XT to a Windows 10 install, if possible.

However, the first order of business is getting the GUI performance at something acceptable so I don’t want to claw my face off (yeah, it’s that bad). I’ve done several searches on Google and in this forum, and checked things like the cpu frequency (xenpm gives a lot of errors trying to read/set CPU parameters). I don’t see anything in the GUI menu for driver settings for X, so I don’t know if it is a driver problem, but I would also like to solve the problem of moving dom0 to the RX550 (secondary) adapter at the same time, if possible.

Any advice? Feel free to link any relevant resources I might have overlooked.

Thank you in advance for any help.

3

Oh, also, The first attempt at getting into the installer was met with a black screen with a non-blinking cursor in the corner. I did some Googling and found a solution that worked by putting “nomodeset=0” on the linux kernel line in GRUB. That allowed the graphical installer to run.

4

You can hide GPU from dom0:
Contents/docs/customization/gaming-hvm.md at master · Qubes-Community/Contents · GitHub

Did you try to install kernel-latest in dom0?

5

If everything works until the point where xorg loads, you can try looking in xorg.conf, could just be that it’s trying to use the wrong device.

6

I did update to kernel-latest, and it is MUCH better now, thanks! No more headache-inducing GUI experience.

Next, I had a few issues getting wired ethernet working. I got WLAN going (which is how I was able to update to kernel-latest), but both of my two ethernet ports kept failing to pull a DHCP address. After fiddling with it for a while, I was able to get it working by disabling IPv6 and setting the MAC address to Permanent. Also, it seems that editing and saving network connection params using the GUI causes adapters to go into a NO-CARRIER state which can never be recovered from until after a complete (power-off) shutdown and restart. So it appears that networking is fixed.

Next, I am trying to get the dom0 to use the Radeon RX550 adapter (which is secondary from POST/UEFI boot time). I tried adding all the kernel settings for pushing the PCI device ids for the RX6950XT into pciback (rd.qubes.hide_pci=04:00:0,04:00.1 xen-pciback.hide=(04:00.0)(04:00.1) xen-pciback.passthrough=1 xen-pciback.permissive) and they do appear attached to the pciback (in use) driver in lspci -k. However, qubes still boots and uses that PCI device, and not the other one (which is PCI device 01:00.x).

I am hoping there’s an additional setting somewhere to tell qubes to use the secondary GPU for dom0? As I mentioned, the motherboard BIOS does not have any kind of setting to specify a boot adapter (probably because there’s no integrated adapter, so they saw no need – typical), and I am not able to swap GPU PCI slots due to mechanical/thermal restrictions in the case.

Thank you again for the assistance! :slight_smile:

7

Are you saying that after Qubes boots it’s still using RX6950XT to output the video even though you can see in lspci that it’s hidden (using pciback driver)?
Or are you saying that your PC is still using the RX6950XT to output the video before Qubes boot but after the boot it’s starting to use RX550?

8

Yes, it is still using the RX6950XT to display the Qubes GUI. Nothing ever displays on the RX550. It doesn’t even get initialized. I know it works in the current configuration because when I first ran the Qubes installer (before I added nomodeset=0 to the kernel line), it displayed some bootup log lines until it hung at X init (I think).

I still have the nomodeset=0 param, though. I didn’t think it would have any effect on this, but maybe I should try removing it and seeing what happens?

9

Are you sure 04:00:0,04:00.1 is RX6950XT and not RX550? I’m not sure how it could work if it’s hidden. Can you post the output of sudo lscpi -vvn for RX6950XT and RX550?

10

Positive.

11

You’re right. Then I’m not sure if it’s possible to hide the primary GPU used by BIOS.

12

Yeah, for the last few weeks prior, I was looking for a way to patch my BIOS to change the GPU selection heuristic. From what I have read, it uses the highest (last?) PCI id as the default, but there has to be a Dxe module which uses a BIOS setting/UEFI variable, since motherboards with IGPUs often have the ability to specify a different display.

That said, I know that the dom0 kernel can output to the other GPU, so there also should be some way to tell it to pick a different one than the default (perhaps something in vgaarb?). I found a kernel patch for vgaarb submitted by someone named Cal Peake some time back which allows you to do just that, but it was never accepted.

I’d prefer to patch the UEFI BIOS so it can use the correct one from the very start, but it still should be possible for the kernel to choose its own.

It definitely should not continue to use a device that’s been “hidden”, but I guess that depends on what “hidden” means, and hidden from whom? I don’t want to disable it, because then I couldn’t use it (unless I could also re-enable it before starting the Windows VM).

13

So, I took out the nomodeset=0 option and, while grub still started on the RX6950XT, once dom0 got to the console lines display, it initialized and displayed the logs on the RX550. Unfortunately, I never get a GUI, as it appears to “hang” at the following lines:

Starting Start Qubes VM sys-firewall…
Starting Start Qubes VM sys-net…
Starting Start Qubes VM sys-usb…
Starting Start Qubes VM sys-whonix…
[ OK ] Finished Start Qubes VM sys-net.
[ OK ] Finished Start Qubes VM sys-usb.
Starting Permit User Sessions…
[ OK ] Finished Permit User Sessions.
[ OK ] Started Command Scheduler.
Starting Light Display Manager…
Starting Hold until boot process finishes up…

and then nothing else happens on the displays. The RX6950XT is blank, but active, and the system is acting like everything is working, but with no GUI.

Maybe I need to tell X to switch to a different adapter in its config, too? This is sorta what happened with the install. X appeared to not start or hang until I added nomodeset=0 to the kernel cmdline.

Still, progress!

14

Update: Alt-F2 brings up the normal text login on the RX550 screen, so I can get in and do things in dom0, so it does seem to be a GUI issue.

eta: It appears X is crashing out because it is missing either config or a driver. I get:

(II) modeset(G0): using drv /dev/dri/card0
(EE) Screen 0 deleted because of no matching config section.
(II) UnloadModule: “modesetting”
(EE) Screen 1 deleted because of no matching config section.
(II) UnloadModule: “fbdev”
(II) UnloadModule: “fbdevhw”
(EE) Screen 1 deleted because of no matching config section.
(II) UnloadModule: “vesa”
(EE) Fatal server error:
(EE) Cannot run in framebuffer mode. Please specify busIDs for all framebuffer devices.

and them some lines pointing at wiki.x.org and the Xorg.0.log file (which is where all this is from).

In the /dev/dri/by-path folder, there are pci-0000:01:00.0-card and -render files, which is the correct PCI device for the RX550. However, the only file in /etc/X11/xorg.conf.d is 00-keyboard.conf. Is the X config in another directory?

15

You can use Xorg :1 -configure to generate xorg.conf

16

Got it. It appears to have created sections for both cards, which is possibly why it’s failing. I’ll remove the Device, Screen, and Monitor sections for the RX6950XT, and remove the extra Screen reference in the ServerLayout section. I notice the driver it specifies is “modesetting”. Is that fine, or should I change it to something related to the card (like “ati” or “radeon”; I presume it is the X driver, not the kernel driver “amdgpu”) ?

17

Update: I went ahead and put the edited xorg.conf file in, and restartd lightdm with “systemcrl restart lightdm” and I have a working Qubes GUI on the secondary GPU! :smiley: :smiley:

Okay, so now I think I am good. I am about to embark on the next problem area of getting a Win10 install with working GPU passthrough of the RX6950XT. I am sure I will have more questions on that subject later, but I think it would be better to start a different topic on that when the time comes.

It does appear that Qubes runs on my hardware sufficiently well. There may be some quirks (I saw some things in the log files related to the CPU and CPU features) I might ask about later, but I will get out of your collective hair for now and do a bunch of reading and fiddling.

Thank you all again for all you help! :slight_smile:

1 Like