Ollama runs smoothly on QubesOS

Community Guides
1

If you like to run an AI model locally this is how I have been running ollama in a dedicated appVM. Performance is alright depending on the size of the choosen model.

Recommended settings for appVM

private storage max size: 80 GB
initial memory: 16000 MB
max memory: what you can spare
VCPUs: 4

In the template:

sudo pacman -Syu
sudo pacman -S ollama
sudo pacman -S docker docker-compose  # optional

In the appVM:

sudo mkdir -p /rw/bind-dirs/var/lib/ollama
sudo mkdir -p /rw/config/qubes-bind-dirs.d
sudo nano /rw/config/qubes-bind-dirs.d/50_user.conf

binds+=( ‘/var/lib/ollama’ )
binds+=( ‘/var/lib/docker’ )

sudo nano /rw/config/rc.local

#!/bin/sh

# increase swap size
swapoff /dev/xvdc1
parted -s /dev/xvdc rm 1
parted -s /dev/xvdc rm 3
parted -s /dev/xvdc mkpart primary linux-swap 10G
mkswap /dev/xvdc
swapon -d /dev/xvdc

# service is disabled in template
systemctl start ollama

# several AI projects offer docker containers, you could
# run ollama in a docker container instead if you like
# systemctl start docker

Restart appVM, download a language model and run it

ollama help
ollama pull llama3.2
ollama run llama3.2

ollama on the command line is used similarly to docker. Using run gives you a chat interface in the terminal, however it’s service also offers an API running/listening on 127.0.0.1:11434. Have fun and may enough RAM be with you.

4 Likes
2

does this guide use PyTorch models? Something to consider…

3

To my knowledge ollama does not support PyTorch models but GGUF only.

You can convert models from the safetensors format (which shouldn’t be vulnerable to deserializing Python object structures) into GGUF in a disposable VM. Also, you can remove the uplink of your VM running ollama whenever you don’t need it.

That being said, Ollama might be vulnerable to binary exploitation. It’s source code is mainly written in Go.

4

ollama is not llama.cpp - still an interesting read

5

Thank you for this guide, it helps !

I noticed that after restarting the AppVM, I have to pull again the model from ollama.
Did I miss some configuration ?

thanks

6

Do I need to passthrough a GPU? because I don’t have one.
And what model would you recommend and how smooth will it run if I will be able to allocate around 10-12GB of ram?

7

Situational. Some models and programs are designed for performing on CPU, but all do much better with a GPU. llama.cpp is one of the projects trying to do that.

Also some models are more computationally efficient than others (in the equivalent size bracket) in general. Of those qwens are really good, but I’m not sure how well it will transfer to your CPU-only setup. As a counter-example, all thinking models are absolute garbage in the efficiency department due to the insane amount of tokens they use for each reply. Speaking of tokens, there are prompt tricks that could speed up the output by lowering token use.

Speed also depends a lot on what CPU you have. With no other context I would say that 10-12 GB is a bit too large for the chat bot kind of use unless you have a really powerful CPU

8

I thought this guide was specifically designed for these who doesn’t have GPU

9

Is Ollama to be install direct on a AppVMQube or on a Template using a repo offline as im sure if i was to install it on a AppVM Qube then each time i close down the said Qube then i would loose Ollama would i not?

10

Ollama installs into /usr/local/bin if you have it in your path, which is persistent on AppVMs.

From https://ollama.com/install.sh :

for BINDIR in /usr/local/bin /usr/bin /bin; do
    echo $PATH | grep -q $BINDIR && break || continue
done
OLLAMA_INSTALL_DIR=$(dirname ${BINDIR})
11

i dont understand anything you say, please kindly make more simple for me to understand?

12

If we check the contents of the ollama installation script, we observe that it checks if you have /usr/local/bin/ in your PATH first among other installation directory candidates.

That means it will install ollama into /usr/local/bin/ if you have it in your PATH. Run echo $PATH to see what you have in path.

/usr/local/ is persistent on AppVMs. Things you install there are kept between reboots and consume private storage of the qube.

The script is not perfect though, it creates ollama user and configures the service in such a way they are not preserved between reboots. It also may try to install drivers if you don’t have them installed already. Service and user are easy to fix by moving /etc/systemd/system/ollama.service to /usr/local/lib/systemd/system/ollama.service and either replacing User and Group values with a persistent user or creating ollama user in the template.