I love Qubes OS for the security and I simply cannot forgo it. But as the local LLMs become predominant, I want to also deploy AI image / video generation models on Qubes OS. I was told that Qubes OS is notoriously insufficient in handling GPU resources (basically Nvidia), as one GPU can be attached to one qube only. Apart from that, is there any other thing to pay attention to? Also, is there any guides for such deployment? Thank you.
The source code and runtime verbosity stacktraces.
I haven’t implemented this setup yet, it’s a side project I plan to pursue once I have all the necessary hardware. But theoretically:
- To preserve Qubes OS’s security features, you should use PVH virtualization.
- The LLM can run on Linux, so you don’t need a Windows HVM qube.
- For the system-level setup, any standard guide for building a gaming qube will work.
- Prefer Debian or Arch Linux over Fedora, as you don’t want to rebuild the qube every six months due to Fedora’s release cycle.
Regarding laptop hardware, that we what for such project:
- Prefer AMD graphics, or find a very well-documented and thoroughly tested NVIDIA solution.
- Consider an external GPU solution, since it makes cooling much easier to manage.
- For an external GPU setup, we need a laptop with at least two sys-usb qubes (or a dedicated port for the graphics card). I think it’s important to isolate the GPU interface in its own sys-gpu qube running as an HVM, separate from other device interfaces.
- As many CPU cores as possible: intel i9 or Ultra 9 series.
- Plenty of RAM.
All of this leads to the bottom line: look for a Qubes OS laptop among devices like:
- ASUS ROG laptops with detachable graphics (though I don’t know whether Qubes OS supports a separate sys-gpu qube for such cards).
- Dell Alienware.