I would like to know how i can download something like Ollama AI LLM direct to a App VM Qube running whatever OS Template you like so that when i shut down the said Qube then restart it that it remains there as i dont want to install Ollama AI LLM to a offline template for obvious reasons plus i dont even think it works that way and just thought it to be easier to download direct to the said qube and use it that way?
There are probably 4+ ways. The main issue is that most of the system paths are owned by the template. So you will need to use a solution that works within in app-vm’s domain.
You could go for an AppImage.
Look at am-gui, lemonade would be a suitable for you.
In that regard, maybe there is a snap/flatpak package … (just a guess, didn’t check)
Otherwise everything you install on /opt is owned by the app-vm. Add it to your path - et voila.
Best option - imho - is using podman/docker. Most things you want are available there. And you could pair it with a systemd user service for startup on vm boot.
archlinux (template)
sudo pacman -S ollama
shutdown
ollama (appvm)
ollama serve
ollama run [your:model]
As long you don’t use the ollama daemon from systemd, all models are downloaded into the user folder and are thereby persistent.
You probably should autostart ollama serve then.
i dont understand?
How do i download Arclinux as i dont see it there in the Template Manager list as even an option?
Have a look at the docs: Templates — Qubes OS 4.3.1 documentation
You don’t need arch to do this, any distribution will do. Many distribute ollama in their package manager, but that makes you use a template to store ollama itself (not necessarily the downloaded models). To store models in AppVM what matters is not the way you install ollama, but how you configure it.
By default ollama stores models in the home directory of the user ollama server runs as. To store models in an AppVM you either need to ensure that home directory is persistent on the AppVM, or directly set where ollama server stores models using OLLAMA_MODELS environment variable.
Probably the easiest way to do this is to set ollama user directory to a persistent location, such as /usr/local/ollama:
# install -d -o ollama -g ollama -m 700 /usr/local/lib/ollama_models
# usermod -d /usr/local/lib/ollama_models ollama
This approach assumes that ollama is installed in a template, and user directory modification requires running the commands in the template.
Alternatively, you could set User and Group of ollama service to a user with persistent home directory, such as your user user, but that is less appropriate unless you plan to use the ollama qube just for ollama and nothing else.
does this mean all i need to do is set up a new Qube based on say; Fedora-44 and issue said qube with say; 80GB presumably then download Ollama using the CLI commands you kindly provided?
I am new to it all but i thought to dedicate Ollama to 1 single Qube and restrict it to only that 1 specific Qube only was my thinking?
IDK how to do persistent anything from a Normal Qube and do i need to put the
export all_proxy=http://127.0.0.1:8082/ command in CLI before i download/install anything to a qube for it to be persistent or is that command only for templates?
To be clear the commands I wrote don’t install ollama, they change home directory of the ollama user, assuming that ollama server runs under namesake user. Thus to use them you need to install ollama in a template first, ensure that ollama server runs under ollama user (in general, not running at the moment), and then run them in the template as usermod doesn’t work in AppVMs.
Installation is up to you, but you probably will end up with either template with ollama installed and AppVM with large private volume, or an AppVM that houses both ollama binary and the models.
For fedora template installation it probably would be something like dnf install ollama. Check what fedora docs say on ollama, maybe there are extra steps I’m not aware of or they use user name other than ollama.
If you prefer to install everything onto an AppVM, you could use installation script from the ollama website. But that has its own disadvantages, for example no semi-automatic updates via package manager.
Regarding persistence, see inheritance and persistence. You don’t need to export proxy address on an AppVM.