[qubes-users] Ollama (or Llama.cpp) with GPU Passthrough

[Metatron]

Dear Qubers,

I would like to enquire if anyone has had any sucess with Ollama with
GPU passthrough?

I am using an Arch linux template that is working well for dedicated
video out and can play media and games (with stutter) which is already a
great convienance on Qubes.

However the main reason I built a passthrough set up was for Ollama /
Llama.cpp.

I've tried pci strict resetting, no dynamic memory balancing, dasharo
and standard bios. Everything works fine if I swap from qubes OS to
Arch.

I'm using ollama-rocm.

I used the gpu-passthrough guide at:

Create a Gaming HVM

Well that’s the funny thing : it’s ok they are showing in dom0, but the VGA part was showing (an is stil lshowing) with nouveau driver when I actually blacklisted it I removed the pci-hide from Grub, but still, nouveau is supposed to be blacklisted and… it seems it still somehow ‘captured’ by the Nvidia card before it is actually blacklisted… Although, a grep nouveau with the dmesg doesn’t show anything… And the BIOS of my laptop contains so few information, even the ctrl+s doesn’…

GPU is detected but loading does not progress beyond:
llm_load_tensors: CPU buffer size = 35.44 MiB
It just hangs forever.

Full dump below.

If anyone has got any further or has any thoughts please let me know!