Create a Gaming HVM

I published it both on my website and in the qubes-community project on Github, you can check in the footer of the original post that I am the author:

"

  • Original author(s) (GitHub usernames): neowutran
  • Original author(s) (forum usernames): @neowutran
    "

everything is ok :slight_smile:

1 Like

I didn’t realize there was such useful info in the details of the migration! :bulb::bulb::bulb:

1 Like

Pretty cool guide! Did someone ever try with a thunderbolt external GPU?

@neowutran I got this working more easily with pipewire. And since there was recent work by @Demi in adding support, I think it’s a good bet.

(Disclaimer: this was on manjaro. Not pure arch)
Here are the instructions:

Getting audio to work

In my case I was having issues with audio. It was stuttering, so I removed the qubes-vm-pulseaudio package and and installed pipewire as a replacement and it seemed to do the job:

sudo pacman -Rdd qubes-vm-pulaseaudio pulseaudio
sudo pacman -S pipewire-{jack,alsa,pulse}

After restarting the machine, the audio was working

2 Likes

I just tried it but it doesn’t seems to work with multiseat Xorg. Do you confirm that for you it work even in the multiseat Xorg ?
( audio work well on my “:0” xorg session, but no sound on my “:1” xorg session. I removed all the configuration/modification related to pulseaudio )

Update:
made it working on my side with pipewire. I needed to use this custom script instead of directly i3 in my example to force connect to the audio daemon

i3.sh 
#!/bin/bash
/bin/sudo -u user PULSE_SERVER=unix:/run/user/1000/pulse/native /usr/bin/i3
1 Like

I just got an HVM to display GNOME on my external GPU :+1:

I did that (I’ll try again from a cleaner environment to get a more reproducible guide)

  • hide the pci devices (there is no IOMMU on the external GPU, except with the AMD card but it’s super clear to see the AMD VGA and AMD audio are part of the same group)
  • pass the devices to an HVM
  • use the qube provided kernel
  • use X :1 -configure to generate an xorg.conf file, then remove all extra in it to just keep the screen/monitor/device related to the VGA card itself was good enough, check you are using nvidia for NVIDIA cards and amdgpu for AMD cards
  • move that file in /etc/X11/xorg.conf.d/99-xorg.conf, and start your display manager, e.g. systemctl start gdm.service

Tested on OpenSUSE tumbleweed with an AMD RX 480 and NVIDIA 1060 :ok_hand:
Tested on OpenSUSE tumbleweed with an NVIDIA 1060 but in discrete mode (display in dom0 but uses the GPU) :ok_hand:

Tested on Debian 12 using the template with an NVIDIA 1060 :ok_hand:

I used a Razer Core X external GPU case connected in thunderbolt 3 on a Lenovo T470, unfortunately it’s almost useless in this case because the CPU is way too slow to do anything meaningful with a GPU in Qubes OS :smiley:

6 Likes

I’m sure using QubesOS doesn’t improve the CPU performance, and mobile CPU are not great for gaming to start with, but do you not also run into some serious PCIe bandwidth issues?

Can you get more than 4x PCIe lanes?, I also think TB3 and 4 only use PCIe gen 3, and it’s not the fast CPU lanes, it’s the slower chipset lanes.

Not really, on Linux I can emulate Switch games using Yuzu or play games like Death Stranding or Control (CPU bound). The laptop has an i5-7300U, it would take a while before being limited by thunderbolt bandwidth. Maybe if you use a 4k screen this would saturate faster?

If you use the eGPU as a discrete GPU, the bandwidth is limiting because the data have to go in both ways, if you use it with an external display, you have more bandwidth because the rendering doesn’t need to go back through thunderbolt.

Yes, it works if you just use it for the external display, but you could do the same with a TB dock without the GPU.

I thought you wanted to connect the GPU and use it to play games that need accelerated graphics.

that’s exactly what I’ve done, and it’s working.

1 Like

Great guide but I still have some questions.

For the Iommu Group do I have to do everything inside the grub cmd of the usb linux live distro (I have to paste the #!/bin/bash part)?
And if yes how can I deal with my OS being LUKS encrypted?

Also, what’s the deal with the max-ram-below-4g?
From what I understood this means this will allow up to 2 GB of ram to your gpu passtrough HVM qube, then I don’t understand how people in other threads have straight up 4090s working. (And what about the VRAM?)

bump, I am also interested

“For the Iommu Group do I have to do everything inside the grub cmd of the usb linux live distro (I have to paste the #!/bin/bash part)?” : No. Boot into any linux live distro, use the standard terminal emulator to create a script and execute it.

“And if yes how can I deal with my OS being LUKS encrypted?”: Irrelevant, you don’t need to access anything on your OS for this step

“From what I understood this means this will allow up to 2 GB of ram to your gpu passtrough HVM qube, then I don’t understand how people in other threads have straight up 4090s working. (And what about the VRAM?)”: No, you have access to all the ram you want and don’t have any kind of limitation on VRAM.

Also, what’s the deal with the max-ram-below-4g?: You can read differents threads here:

or search on internet. Not everything is understood about what is going on with TOLUD

2 Likes

Btw, I did a presentation about my video editing setup on Qubes, which has GPU passthrough on an nvidia 4090, if anyone is interested.

Timestamp 5h34m30s

10 Likes

The lines for the original post on patching xen.xml has changed because the original patch interfered with audio VM’s.
Maybe update the lines and link to the thread in case more progress is made?

I am going to remove the information about xen.xml patching from this guide.

  • The two method (xen.xml / stubdom-linux-rootfs.gz ) are currently not doing the exact same things
  • The difference between the two methods seems to be causing confusion / I see lot mistakes in forum posts ; and it make troubleshooting harder
  • I don’t personally use the xen.xml method and I am not willing to spend time updating xen.xml patching method to have the exact same behavior as the other method
3 Likes

Are you still having to keep stubdom downgraded to apply the stubdom patch or did they resolve that?

I am using the latest version of everything available in the official repositories

I don’t understand what this means.

I want to be able to use a GPU in order to learn about open-source LLMs. It looks like many of those on git are Windows only.

Why do you recommend using a Linux guest? I also don’t know what “Linux Guest” means. Does this mean a graphical environment like what you would see if you install directly from an ISO into a StandAlone VM and not a template?

There is also mentioning creating a Standalone Qube based on your choice of template. I think only Pop!OS has NVIDIA drivers built in and I have never heard of anyone using Pop!OS inside Qubes.

I know this is an advanced topic for those with advanced linux backgrounds. I am going to try to do this despite not being advanced and I will hope I do not brick.

I have an extra monitor and an extra mouse and keyboard. Is the best way to go about this to follow this guide? Pop!OS is so easy but I feel very unsafe when I am not using Qubes.

Why do you recommend using a Linux guest? I also don’t know what “Linux Guest” means. Does this mean a graphical environment like what you would see if you install directly from an ISO into a StandAlone VM and not a template?

“Guest” as the opposed of “Host”. You could call it a “linux qube” or “linux vm”.
I recommend using a Linux guest because it is easier to debug/modify/patch than Windows.
And is integrated with QubesOS, better supported and better performance than Windows guest

There is also mentioning creating a Standalone Qube based on your choice of template. I think only Pop!OS has NVIDIA drivers built in and I have never heard of anyone using Pop!OS inside Qubes.

You can install nvidia driver on any distribution (debian, fedora, archlinux, …).

I have an extra monitor and an extra mouse and keyboard. Is the best way to go about this to follow this guide? Pop!OS is so easy but I feel very unsafe when I am not using Qubes.

Only you can answer this question. I am able to run ML models using the setup described in this guide. If you want to follow this guide, it is up to you.