This is not a conventional HCL report, as I have the - very opportunistic - idea to create a ‘gaming VM’ on Qubes OS
I’m not really believe if it’s gonna be a success story, but I decided to give it a try before downgrading my expectations and sliding towards a KVM based solution - or as a last resort: going back to a native Windows gaming rig, what I currently have (next to my everyday Qubes Laptop) but really really want to get rid of.
I will update this topic during the ‘project’ to collect and share the results…
There are no PS/2 ports, so all of my imput devices - including the keyboard - must use one of the many USB ports. And this is already a security compromise.
USB-C in DP alt mode, would be nice if if would work - but I would not bet on that - and also a possible attack surface.
BIOS Flasback, what is advertised as a nice and user friedly feature, but I see it as a huge attack surface instead
So this is not a good start, but those are the risks I’m willing to accept in case of a desktop PC located in my private property…
And frankly, I would be proud of my kids if they would even try to hack my PC usign those very open attack surfaces
Memory slots: the marketing vs. reality
The board has 4 slots, which looks very niece. And the CPU is also supports four memory slots too… but as it was turn out - in the hard way - if you fill all of those, the RAM modules are drastically downscaled!
I initially bought 4x DDR5@5600MHz modules, that was downclocked to 3600MHz
So in order to get reasonable RAM speed, you must only utilize 2 memory slots only! Luckily it was still feasible to reach the target 64GB of RAM, by a 2x32 Kit.
The next challenge is to ‘get’ a nice and usable IOMMU groupings… But yes, you have nothing to do about this, but to find out what the vendor providing for you. So it is a pure gamble, unless others already tested the very same hardware for you
As you see, there are ‘nice’ groups, which might allows me to successfully pass trough them to different VM’s.
However I noticed that as you insert more and more PCI devices, the PCI ID’s, and the groupings are also changing! Yes, that means the goup numbers are depending on how many PCI devices you have… But at least the go together devices are remaining the same. So if your dGPU is alone in a group - like in case of this board - it will be always alone, but under a different ‘group number’, and might be with a different PCI ID!
Yes, this means if you already assigned some PCI devices to a VM, then insert a new nvme SSD - for example - then your VM’s will be messed up badly. And might will not start in practice. That’s combined with USB keyboard troughth sys-usb is a fatal error. You need to poke your grub config to fix such issue…
So, it is wise to install all your (PCI) devices (SSD, VGA, etc) before you even try to install Qubes on this machine.
If you ‘finally’ reach this point, then still comes more disappointment As the installer is bugs out badly, leaving you a half baked system.
As many of you probably already know: debugging such issues is a pain in the ass. So I was just rebooted the PC, and surprise surprise: a Qubes boot screen was appeared on my screen
So I let it boot up, but the ‘first install’ screen was bugged out too.
Yet another reboot: And as I guessed, my keyboard was not functional after I successfully entered my LUKS password. Means the sys-usb creation was failing somehow, but at least was causing my keyboard issue.
So I had to stop the sys-usb autostart, to be able to see if I can login to the system…
And for my biggest surprise: I was able to login whit this ‘hack’ - but ofc this still needs a real fix instead of this ‘dirty’ workaround… which is on my TODO list now.
manually started the sys-net: and yee-haa, I have internet connection! So it is my first success
tried to ‘sleep’ the machine, which is half success or failure?!: As it was went to sleep successfully, but was never wake up - and this is where I’m right now. → this is also on my TODO list.
Yes, I do already have a HCL report too, what I will upload for sure… but I think we all agree that would be very misleading without these details, right?
I think this is normal for AMD Zen4/5 systems, they have multiple USB controllers and one of them can’t be passed to sys-usb.
If you want to automatically create sys-usb during installation, the system will crash then creating sys-usb. You have to reboot with qubes.skip_autostart, to complete the installation, and when in Qubes OS remove the problematic controller from sys-usb.
Once you have set up sys-usb correctly, everything should work just fine.
It avoids getting locked out when sys-usb is dead, but it can mean that a lot of the external USB ports become unusable (except for kB/mouse). I have a lot of tape over unusable ones to avoid wasted time, as there’s no popup from Dom0 when an invalid device is refused, after accidentally plugging it to one of them.
Edit: it can also stop working if the BDF of the controller changes when other PCI hardware devices are added
I was able to fix the USB assignments, by manually removing the automatically added devices, and re added all my USB Hosts. surprisingly there was no issue this way, so I don’t know why the installer created sys-usb was not working…
However, the non-working sleep function is a showstopper
As there are no logs related to sleep, so I’m stuck and without ideas how to fix this…
The symptom:
The machine newer wakes up, and CPU and RAM error leds remaining lit.
(on a normal boot all the leds are lit up for a few sec in a specific order, but then goes off before reaching the UEFI screen)
As the same build works with Ubuntu 24.04 LTS kernel 6.14 - so it must be xen/Qubes related issue.
I have successfully ‘hiding’ the VGA (and the corresponding audio device) and when I adding it to a windows VM, I see the devices in the VM, but no signal on the directly attached display
If I install the nvidia drivers inside the windows VM, then it is successfully showing my VGA model, but automaticall disabling it with the ‘Code 43’ error status. (manually disabling/enabling also not helps)
I have tried the permissive and no-strict reset parameters, but neiter of those are making any difference and I see no errors in dom0 (or I don’t know ehere to look)
I have tried with windows 10 LTSC and 11 LTSC guests - with the very same resuts.
The only difference I found if I do not install the QWT, then after attaching my VGA the windows boots, but wors like a slide show with 0.5 FPS.
(and still nothing on the directly attached display)
QWT in general: seems to be working, exept the GUI daemon.
If I install that module, all the window drawings are breaks. Technically renders the VM unusable.
just as described here:
The verdict
As I had no luck with Qubes regarding VGA passtrough, I have tried this on Ubuntu (24.04 LTS) with kvm, qemu, virt-manager and Win 11 LTS IoT as a guest OS.
The experience was a way more smoother… and more importantly I was able to successfully create a working gaming VM.
Where working means I can replace my previously used native windows for a more ‘sustainable’ Gaming (and Photo-workflow) rig.
The key differences I have noticed, and might related to the success:
UEFI boot for the VM,
the ability to disabling the ‘ROM BAR’ for the VGA that I passed to the VM
And neither of those seems to be available in Qubes, or it might needs some undocumented tweaks to achieve, that I’m not aware of…
The other smaller things, that makes me stay on Ubuntu with KVM for now:
the working virtio-win drivers vs. the beta stage of the QWT,
easy to create and restore VM snapshots,
The ability to calibrate my screen (not for gaming, but for my photography workflow)
As I have already dedicated significant time for this project, and I know that my current hardware is surely usable for this… I might try again later.
- and also a possible attack surface.
