Best way to use GPU for AI models on Qubes

[PostedPortal]

Hi,

I saw some guides that show how to utilize GPUs for gaming qubes. However, its not so straightforward, may not work on some setups, and most importantly there are heavy performance drawbacks.

My use cases relies on running custom LLMs locally. The data cant leave my house. It must be used under the same network

Would i be better off setting up a local AI/GPU server?
Would using my thunderbolt 4 port to hook up an external GPU work better?

[apparatus]

There are guides for AI as well:

Running local LLMs with or without GPU acceleration

This guide explains how to install text-generation-webui (oobabooga) on Qubes OS 4.2.0, it can be used with nvidia, amd, and intel arc GPUs, and/or CPU. I will only cover nvidia GPU and CPU, but the steps should be similar for the remaining GPU types. The GPU used is the nvidia 4060, it might not be exactly the same for nvidia GPUs that use the legacy driver. Not having a GPU is going to greatly limit the size of model you can use, and even small models are going to take relatively long to exe…

Running local txt2img with stable diffusion

This guide explains how to install stable-diffusion-webui (AUTOMATIC1111) on Qubes OS 4.2.0. It’s meant to be used with an nvidia GPU, but it can run on the CPU. The GPU I use is the nvidia 4060, I’m not sure if it will work with cards using the legacy driver, I’m also not sure how to make it work with other GPU types. I have tested SD with and without using a GPU on 3 systems running Qubes OS, and it does work only using the CPU. So if you just want to try SD before you go out and buy a $1000…

@PostedPortal

Would using my thunderbolt 4 port to hook up an external GPU work better?

Work better than what? Than the dGPU inside your PC?
If it’s the same model of GPU and if it’s not overheating then no.