Why does seamless GPU acceleration only work when using template based HVMs?

[renehoj]

Here is a screenshot of Cyberpunk 2077 running in window mode with full GPU acceleration, I only get 25 FPS on high settings but the GPU I use is only a 4060.

Why does this only work when I use a HVM based on a template?

If I use any other type of HVM I can’t get it to work, I tried different Linux ISO/distributions, different nvidia drivers, I tried running the HVM appVM in debug mode with gui-agent removed, nothing works.

Is there any way I can do the same with a HVM that is exposing the entire desktop?

Most/all games that can run with Wine+Proton seem to work, but many games don’t work well with the Qubes OS virtual screen and input, which is why I’m trying to make it work with a HVM running a full desktop.

[apparatus]

@renehoj

I tried running the HVM appVM in debug mode with gui-agent removed, nothing works.

You don’t see the HVM desktop after this or you see it but GPU acceleration is not working?
If you don’t see the desktop then did you set the default target after removing qubes-gui-agent?

sudo systemctl set-default graphical.target

You can also try to use VNC:

Automatic startup of Gnome environement

I think this is not the best approach. It produces a standalone and it removes one of the things that Qubes has worked hard to provide - the functionality between qubes. Do this instead - In the template you want to use, install tigervnc-standalone-server and tigervnc-viewer. Create a script at /etc/skel/gnome.sh with this content: #!/bin/sh tigervncserver -localhost -xstartup /usr/bin/gnome-session -geometry 1024x768 -SecurityTypes None -useold :1 && xtigervncviewer 127.0.0.1:5901 & Ma…

[renehoj]

I see the desktop, and the GPU get detected, but the performance is a lot worse.

I’m guessing some driver stops working as soon as I stop using the virtual display driver that only shows the application and not the desktop.

I’ll try using the VNC and see if that works.

[renehoj]

VNC partly works, the GPU acceleration works, and it allows games to use native full screen. Sadly, it seems to have issues if the screen/view port gets locked to the mouse.

[apparatus]

I don’t understand what issue do you have from your description, but if it’s something related to the mouse input then you can try this:
Qubes redirect mouse input to VM · GitHub

[Quben]

Wait what…? GPU accelewho? When did y’all start gaming in qubes with GPUs and didn’t invite me???

[renehoj]

In some games, free camera motion is done by locking the mouse to the center of the screen, and running a continues loop that reads mouse movement.

In some games when using the VNC, the mouse doesn’t get locked or the position of the mouse is not read correctly, and the result is that the camera spins uncontrollably when you move the mouse.

The input proxy doesn’t work and attaching a USB controller also doesn’t work, but that could be a configuration issue in the VM and the input are not working in display localhost:1

[apparatus]

@apparatus

@renehoj

I tried running the HVM appVM in debug mode with gui-agent removed, nothing works.

You don’t see the HVM desktop after this or you see it but GPU acceleration is not working?
If you don’t see the desktop then did you set the default target after removing qubes-gui-agent?

sudo systemctl set-default graphical.target

I’ve tried it and it worked for me. Try like this:

export VGL_DISPLAY=egl
vglrun glxinfo | grep render

@apparatus

I don’t understand what issue do you have from your description, but if it’s something related to the mouse input then you can try this:
Qubes redirect mouse input to VM · GitHub

Also this worked for me, I’ve tested it in minetest and run this command in dom0 to attach the mouse from sys-usb to my gaming qube:

qvm-run -u root --pass-io --localcmd="qvm-run -u root --pass-io sys-usb \"input-proxy-sender /dev/input/by-id/usb-MY_MOUSE_ID-event-mouse\"" MyGamingQube "input-proxy-receiver --mouse"

[apparatus]

I’ve tested it some more and I’ve found the problem that you’ve encountered, it was Xorg. Using Wayland don’t have this performance drop.
Here is an example with sway, but it should work for KDE/GNOME/etc with Wayland if you prefer it:
Install sway in the template (e.g. sudo apt install sway xwayland for debian).
In the gaming app qube based on this template create the custom sway config:

mkdir -p ~/.config/sway
cp /etc/sway/config ~/.config/sway/
nano ~/.config/sway/config
echo "seat * pointer_constraint disable" >> ~/.config/sway/config

Attach mouse to the gaming qube by running this command in dom0 or attaching the USB mouse directly to the qube:

qvm-run -u root --pass-io --localcmd="qvm-run -u root --pass-io sys-usb \"input-proxy-sender /dev/input/by-id/usb-MY_MOUSE_ID-event-mouse\"" MyGamingQube "input-proxy-receiver --mouse"

Start sway in the gaming qube’s terminal:

WLR_NO_HARDWARE_CURSORS=1 sway --unsupported-gpu

Run the game inside the sway.

[apparatus]

I spoke too soon, the mouse is not working properly with Qubes GUI agent.
The seat * pointer_constraint disable stops the mouse from spinning but it’s limiting the mouse movement.
Also passing the mouse input using input-proxy seems to only work in Xorg, it’s not working in Wayland so you’ll need to pass the USB mouse to the qube.
Running sway without Qubes GUI agent in the HVM stubdomain window works fine:

Enable debug mode for gaming qube and gaming qube’s template in dom0:

qvm-prefs MyGamingQube debug 1
qvm-prefs MyGamingQubeTemplate debug 1

Increase the gaming qube VRAM for stubdom, the default value didn’t work for 1920x1080x60Hz mode:
Create this file in dom0:

nano /usr/share/qubes/templates/libvirt/xen-user.xml

And add this content to it:

{% extends 'libvirt/xen.xml' %}
{% block devices %}
        {{ super() }}
            {% if vm.features.get('increased_vram', '0') == '1' -%}
                <video>
                    <model type='vga' vram='32768' heads='1' primary='yes'/>
                </video>
            {% endif -%}
{% endblock %}

And enable the increased_vram feature for your gaming qube in dom0:

qvm-feature MyGamingQube increased_vram 1

If you use kernel provided by qube then in gaming qube template add the missing kenel options compared to the kernel provided by dom0, e.g. for debian:
Add this line at the end of the /etc/default/grub in the template:

GRUB_CMDLINE_LINUX_DEFAULT="$GRUB_CMDLINE_LINUX_DEFAULT rd_NO_PLYMOUTH rd.plymouth.enable=0 plymouth.enable=0 apparmor=1 security=apparmor"

And regenerate the GRUB config, e.g. for debian:

sudo update-grub

Plymouth is messing up the HVM console so you need to disable it.

Remove the qubes-gui-agent from the template, e.g. for debian:

sudo apt remove --purge qubes-gui-agent

Shitdown the gaming qube’s template.
Enable debug mode for gaming qube:

Start gaming qube, login as user user and create a custom sway config:

mkdir -p ~/.config/sway
cp /etc/sway/config ~/.config/sway/
nano ~/.config/sway/config
cat << 'EOF' | tee -a ~/.config/sway/config
output Virtual-1 {
    mode 1920x1080@60.000Hz
}
EOF

You can change the display mode from 1920x1080@60.000Hz to another one.
You can check the available display modes by running this command inside a terminal in sway:

swaymsg -t get_outputs

Run sway:

sway --unsupported-gpu

Then run game in the sway.

UPD:

The performance when running full desktop in HVM stubdomain window is still lower compared to running the full desktop with Qubes GUI agent.
This seems to be because the CPU is copying the image from the qube which slows things down.
When using Qubes GUI agent the image from the qube is directly copied to dom0.
When using HVM stubdomain window the image from the qube is first copied to the stubdomain qubename-dm and then copied from stubdomain to dom0.

@renehoj

Most/all games that can run with Wine+Proton seem to work, but many games don’t work well with the Qubes OS virtual screen and input, which is why I’m trying to make it work with a HVM running a full desktop.

I’ve encountered the problem with Wine+Proton similar to this one:
Fullscreen apps with wine unusable · Issue #7307 · QubesOS/qubes-issues · GitHub
And suggestion to enable virtual desktop feature in WIne using winecfg helped and fixed the problem with games for me. E.g. using the Wine from steam’s Proton:

WINEPREFIX=/home/user/.steam/steam/steamapps/compatdata/$APPID/pfx ~/.steam/steam/steamapps/common/Proton\ -\ Experimental/files/bin/wine64 winecfg

Change $APPID to the ID of the steam app.
Change Proton\ -\ Experimental to your version of Proton if it differs.
In the opened winecfg enable:
Graphics → Enable a virtual desktop
And set desired resolution.
Viewing topic Proton: Enabling Virtual Desktop and disabling mouse grabbing | GamingOnLinux

[renehoj]

@apparatus

When using Qubes GUI agent the image from the qube is directly copied to dom0.

It’s the modified xf86 dummy driver, which doesn’t seem to work without the qubes input driver. When I realized it wasn’t possible to get the same performance without using the Qubes OS drivers, I kinda gave up making hybrid graphics work.

Sway doesn’t work because prime is active, VNC kinda works but doesn’t have enough bandwidth. Passing a mouse to the appVM works, but you can’t see the cursor, and not passing the mouse turns the game window/screen white when you click the mouse.

Maybe it works better with VirtualGL, but I wanted to better performance from hybrid graphics.

[apparatus]

@renehoj

Sway doesn’t work because prime is active,

What issue do you have with it?
I have a laptop with hybrid graphics as well, Intel GPU in dom0 and NVIDIA GPU in gaming qube.
I’m starting sway without using GPU and games started in sway without VirtualGL are rendered using Vulkan with GPU and work fine.