What Is the Current State of Qubes OS GPU Support?

General Discussion
1

I am a complete newbie on the technical side. However, I do know that more and more applications require hardware acceleration and that QubesOS could be much more powerful if GPU support were given. At the same time, GPU usage seems to be a significant additional attack surface that should not be taken lightly. However, my perspective on this is that a theoretically increased attack vector on QubesOS with GPU support is still a far better option than moving to much more secure alternatives.

Both arguments are valid and just depend on the threat model of the specific user. Obviously, if you need the highest level of protection possible, you should not use a GPU QubesOS system. But if you need a GPU application, it would still be the best option.

Recently there was news about GPU KVM VirtIO and AMDGPU Mesa support. Would something like that work on QubesOS?

https://www.phoronix.com/news/AMDGPU-VirtIO-Native-Mesa-25.0

2 Likes
2

GPUs are well supported in Qubes for my use case via PCI passthrough. The attack surface isn’t a major concern for me due to the role of the VMs that utilize the GPU

It’s not immediately clear to me what the benefits are to using the approach described in the link you provided but it sounds like it provides the VM some abstraction of the PCI device, without fully abstracting a powerful GPU to a “dumb” VGA device in the guest. So it can use the native drivers without interacting directly with a PCI device

If so, I imagine it would probably reduce attack surface as it would just be shims for the GPU driver in the guest to interface with the GPU rather than having direct access to a PCI device. But I’m really not too sure I’m understanding exactly what the technology is

It’s interesting, but I’m very happy with what I have now. Interested to hear others’ thoughts on this

2 Likes
3

I have read that a GPU can only be used by 1 system at a time. This is not something that QubesOS decided, it’s the way hardware works and is same on all operating systems. That’s why you need an internal gpu for dom0 and a dedicated gpu for the qube.

I took these notes from qubesos forum but i can’t remember who said it:

Make sure to buy a MUXed laptop: [GUIDE] Optimus laptop dGPU passthrough · GitHub
So you’ll be able to directly connect your dGPU output to the external display.
Even with MUXless laptop, you can still use dGPU to render the graphics in the qube and then the rendered image will be copied from the qube to dom0 using CPU to display the qube’s image on dom0 screen but the performance will be poor compared to outputting the image directly from the qube to the dedicated display.

Approximately 30% less performance because of the bandwidth requirement for the render stream.

You can only be sure that the laptop is MUXed if it was reported by someone that bought the laptop and checked it or if the vendor specifically stated it.
For example, DELL states if the laptop supports the Direct Graphics Controller Direct Output mode on their website.

1 Like
4

See also:

1 Like
5

I guess it depends on your hardware.

In my personal experience, GPU support isn’t great, but it’s also not terrible.

I’m using 2 GPUs in my system to get around the issue with only one VM being able to use the GPU, I rarely need acceleration in more than two VMs at the same time.

I have 1 GPU dedicated to providing AI integration for VMs, using local LLMs through the Ollama API. It works reasonably well and allows local AI integration in web browsers, VS Code, etc. without having to have a GPU attached to the VM running the application.

The other GPU I use for gaming and productivity application that require GPU acceleration, which will be limited to one active VMs at a time.

In my personal experience, I don’t find a current state of GPU support is preventing me from using the applications I want to use, or make it significantly more difficult to use them.

3 Likes
6

This is great to hear, and this is also my future usecase for a @novacustom laptop with rtx4070 dGPU. Do you have a guide somewhere in the forum for setting the dGPU up with a debian qube for running local LLMs and Image Generator AI models (flux + comfyUI)?

Awesome. I am also looking forward to be able to game on a windows qube with no network connectivity and run all the https://fitgirl-repacks.site rips :smiley: safely and isolated

3 Likes
7

Nice idea and set up! Could you share exact specs of your hardware/firmware?

2 Likes
8

I’m using stock firmware.

CPU: AMD Ryzen 9 9950X
Motherboard: MSI MAG Tomahawk X670e
GPU: 2x Nvidia 4060 Ti 16GB
RAM: Gskill 96GB (2x48) 6800MT/s

1 Like
9

I apologize in advance, I’m going to take this off-topic, into general talk and specific speculation about SR-IOV… bear with me, if interested

I’m not very knowledgeable about SR-IOV, and strictly speaking what you’ve said is correct - one domain can’t share a GPU for display, based on what I’ve read as well

But if I add a pedantic qualifier to what you said, and butcher your words up a little more, I can use it as an excuse to speculate/ask about SR-IOV and reduction of attack surface

GPU can only be used for display purposes by 1 system^H^H^H^H^H^H VM a time

What I’m alluding to is that some GPUs do support sharing the resources of one physical PCI device to many VMs via creation of multiple SR-IOV Virtual Functions

My understanding of VFs is that they’re logical PCI devices, represented with the same PCI BDF, except each with their own BDF function number

I believe VFs are considered effective as security boundaries when configured correctly, which makes them relevant to Qubes.

Aside from facilitating sharing of GPU resources across multiple domains, I’m wondering if there may be security benefits by creating and passing through a VF of a GPU to a single VM rather than the “full” physical PCI device. Maybe that would improve attack surface? Using it only for that purpose, not necessarily sharing it across multiple VMs

The well-known limiting factor of SR-IOV is that the hardware (NIC, GPU, NVME) must explicitly support SR-IOV, and the CPU as well. I have a NIC supporting SR-IOV, but not GPU. And there are only a small few NVME drives supporting it currently

Hopefully someone finds this topic interesting because I’m very far off topic now. And, once more, I’m far from an expert on the topic, so any pros please correct where I have it wrong. And add what you know if you don’t mind

More towards the topic of Qubes, specifically, because VFs appear as their own PCI device, I’m (maybe naively or incorrectly) assuming that if the device and libvirt can do it, Qubes can do it. It may not be straightforward or convenient, but I think it’s doable on Qubes. For sharing GPU compute, NICs and NVME

I’m mainly interested in creating a single VF for my NIC and passing the VF through to sys-net, if it effectively reduces attack surface

Ditto for NVME and GPU, but neither of mine support SR-IOV :disappointed:

Back to regularly scheduled programming :blush:

Some related posts

Interesting Qubes forum post about NIC VFs
Off-site post about libvirt and SR-IOV (on KVM)
Libvirt docs on SR-IOV networking

… I’m clearly not the first one to discuss SR-IOV on Qubes. I may be the one to have most recently discovered it, though

2 Likes
10

Do you mean getting a dGPU’s VRAM to be divided in between a few qubes? Similar to how the available hardware RAM gets allocated between many qubes?

That would be great for qubesos usability; if the discrete GPU resources were divisiable and provisionable like so.

1 Like
11

If it works in Qubes OS, don’t know if anyone has ever tested it.

The cheapest somewhat modern GPU you can get that support SR-IOV is something like that Nvidia A2, which costs around $1500.

You are getting close to half the performance at 4x the price, compared to a similar consumer board without SR-IOV.

1 Like
12

Because my GPUs don’t support it, I haven’t looked into the details of how exactly GPU resources (compute, VRAM) are split up with SR-IOV. Maybe it’s as simple as total compute / num_vfs and total vram / num_vfs, split equally for each VF

I’ve only read details of NICs (simpler) and NVME (tightly coupled to NVME namespaces)

1 Like
13

I own and prefer to use NVidia GPUs and what you’re describing (cost per performance) is exactly why I don’t bother seeking out GPUs with SR-IOV support

however, I think AMD has cheaper GPUs that support SR-IOV, and Intel definitely does. I think even my cheap Intel Arc A310 supports VFs (<$100) though I only use it to drive dom0 display, no acceleration or compute

1 Like
14

Has AMD produced any SR-IOV GPUs in the last 10 years?

The only cheap option, if you have the CPU, I’ve found is the 12th gen Intel CPUs, its internal graphics has support for SR-IOV. Sadly, it was the only generation with that feature.

1 Like
15

I thought it was more common with modern AMD GPUs but seems I imagined that. It really is only newer Intel that are affordable, as you said. The newer internal GPUs as well as some of the Intel Arc

There’s a matrix here though I’m not sure how comprehensive it is

Being limited to mostly datacenter cards does make it seems like it probably won’t be used widely within Qubes any time soon, if ever

Makes sense, it’s a pretty niche feature for a workstation GPU

EDIT: and I may be wrong about the Intel Arc line… seems there’s a firmware hack to do it on some models? Further reinforces the point

1 Like
16

@Churros As far as I understood this information, the benefit is a huge improvement of performance in the VMs. A cite from that link:

"One of the most exciting takeaways from this work:

"With the current protocol Unigine Heaven and Superposition are more or less running at 99% the host speed.""

For me, it looks like a poor man’s “SR-IOV” for GPU. Since GPU manufacturers do not and will not implement SR-IOV in consumer cards, GPU SR-IOV is not on the table for foreseeable future. If this improvement really allows to obtain near-native graphics performance in VMs, then AMD GPUs look like a great choice for such use cases

2 Likes
17

… ignore, got confused about threads …

2 Likes
18

Why do you give VMs GPUs? Are you happy with any GPU or does it have to be a specific brand/model? I have some knowledge of KVM GPU passthrough. How is this different with Qubes/Xen?

How do you use your GPU? For high frame rate rendering on a second monitor or for internal computing without any display (dummy plug)?

1 Like
19

How exactly do you expose the Ollama API to other VMs?

1 Like
20

Check out SwarmUI. It comes with Comfy in the background and has FLUX support.

1 Like