GPU Passthrough

[apparatus]

Running local LLMs with or without GPU acceleration

This guide explains how to install text-generation-webui (oobabooga) on Qubes OS 4.2.0, it can be used with nvidia, amd, and intel arc GPUs, and/or CPU. I will only cover nvidia GPU and CPU, but the steps should be similar for the remaining GPU types. The GPU used is the nvidia 4060, it might not be exactly the same for nvidia GPUs that use the legacy driver. Not having a GPU is going to greatly limit the size of model you can use, and even small models are going to take relatively long to exe…

@anon42499160

and while writing I recognized is it probably not a good idea to have noveau in dom0 and the properitary driver in the qubes, so i will have to stick to one of them, right?

It doesn’t matter since dom0 won’t use this GPU because it’s passthrough to another qube.