Qubes Salt Beginner’s Guide
Part 1: Creating our first qubes
As a beginner, Salt seemed daunting to me at first. It took some effort to learn but it was worth it! I’m writing this guide for beginners who enjoy an hands-on introduction with examples.
1.1 Creating personal state configuration directories
Our journey starts with a file found in the base Salt configuration directory in dom0: /srv/salt/qubes/README.rst (GitHub link). In this file we can read:
qubes.user-dirsInstall and maintain user salt and pillar directories for personal state configurations:
/srv/user_salt /srv/user_pillarUser defined scripts will not be removed on removal of qubes-mgmt-salt by design nor will they be modified on any updates, other than permissions being enforced.
We can activate qubes.user-dirs to create personal state configuration directories. What is qubes.user-dirs, and how do we activate it? This is what we call a state configuration. It is a configuration file that tells Salt what to do to reach a particular state.
To activate qubes.user-dirs, we can follow the instructions found in its configuration file, /srv/salt/qubes/user-dirs.sls (GitHub link):
qubes.user-dirs =============== Install and maintain user salt and pillar directories for personal state configurations: Includes a simple locale state file User defined scripts will not be removed on removal of qubes-mgmt-salt by design nor will they be modified on any updates, other than permissions being enforced. Execute: -------- qubesctl state.sls qubes.user-dirs
We run the command sudo qubesctl state.sls qubes.user-dirs. Salt applies the corresponding state, and tells us that some files and directories were created. Among these directories we can find /srv/user_salt/: this is the main directory where we’ll place our own state configuration files.
1.2 The top file and running highstate
Running the state qubes.user-dirs also created the file /srv/user_salt/top.sls. Here is what this file looks like before we modify it (GitHub link):
# vim: set syntax=yaml ts=2 sw=2 sts=2 et : # # 1) Intial Setup: sync any modules, etc # --> qubesctl saltutil.sync_all # # 2) Initial Key Import: # --> qubesctl state.sls salt.gnupg # # 3) Highstate will execute all states # --> qubesctl state.highstate # # 4) Highstate test mode only. Note note all states seem to conform to test # mode and may apply state anyway. Needs more testing to confirm or not! # --> qubesctl state.highstate test=True # === User Defined Salt States ================================================ #user: # '*': # - locale
This file is called the top file.
In the future, when we have many state configuration files, it will become quite tedious to run each state one by one with the command sudo qubesctl state.sls my-custom-state. The top file solves that. If we write in this file how to run each state, we get the ability to run all of them with a single command: sudo qubesctl state.highstate. We call this “running highstate”.
1.3 Targeting qubes
There are three lines that are commented out at the end of the top file /srv/user_salt/top.sls:
user: '*': - locale
If we were to uncomment those lines and run highstate, Salt would run in all targeted qubes (this is what is meant by the * character) the state locale, for which the state configuration file can either be /srv/user_salt/locale.sls or /srv/user_salt/locale/init.sls.
How do we target a qube? By default, the commands qubesctl state.sls my-custom-state and qubesctl state.highstate only target dom0. To make Salt target additional qubes, we can give their names to the --targets argument:
sudo qubesctl --targets=fedora-38 state.sls my-custom-statewill runmy-custom-statetargeting dom0 and fedora-38.sudo qubesctl --skip-dom0 --targets=debian-12,untrusted state.highstatewill run highstate targeting the qubes debian-12 and untrusted but not dom0.
1.4. Creating a qube with Salt
We have a template called fedora-38. We would like Salt to create a purple qube named “salty” based on this template. We write the state configuration file /srv/user_salt/salty.sls as follows:
salty--create-qube:
qvm.vm:
- name: salty
- present:
- template: fedora-38
- label: purple
- prefs:
- label: purple
That’s it! Running sudo qubesctl state.sls salty saltenv=user will make Salt create a purple qube named salty. If salty is already present, Salt will just make sure it’s purple but won’t do anything else.
Note on "saltenv=user"
Note that we always need to add the extra argument saltenv=user to the command sudo qubesctl state.sls my-custom-state when we run individual states from the user directory /srv/user_salt/.
To make things easier, we would like to automatically run this state when we run highstate. We add the following to the top file /srv/user_salt/top.sls:
user:
dom0:
- salty
Great! Now, the command sudo qubesctl state.highstate will automatically create salty.
1.5 Creating a disconnected qube
We have a template called debian-11. We would like Salt to create a green qube named “disconnected” based on this template, but that has no web browser and no internet access. We write the state configuration file /srv/user_salt/disconnected.sls as follows:
disconnected--create-qube:
qvm.vm:
- name: disconnected
- present:
- template: debian-11
- label: green
- prefs:
- label: green
- netvm: none
- features:
- set:
- menu-items: org.gnome.Terminal.desktop org.gnome.Nautilus.desktop
Perfect! We can now make Salt create this qube with the command sudo qubesctl state.sls disconnected saltenv=user.
To make things easier, we would like to automatically run this state when we run highstate. We add the following lines to the top file /srv/user_salt/top.sls:
user:
dom0:
- disconnected
Great! Now, the command sudo qubesctl state.highstate will automatically create our disconnected qube.
Tip: How to make Salt create both "salty" and "disconnected" when we run highstate?
We can write the top file /srv/user_salt/top.sls as follows:
user:
dom0:
- salty
- disconnected
1.6 Useful links
I hope this was clear. Here are some links if you’d like to go further:
- Qubes docs on Salt
- Qubes configuration management, extensive docs by @gonzalo-bulnes
- Salt user guide, a complete overview of Salt targeted at beginners
- Salt docs, great when looking for something specific but intimidating to start with
- @unman’s notes, a good reference on Salt written by a developer of Qubes OS
- qvm commands, list of all qubes-specific salt commands and what they do
- Qubes state configuration files used to create the first qubes when installing Qubes OS
The next part of this guide will be about creating new templates and installing packages in them.
Part 2: Apps and templates
In this part we’ll learn how use Salt to make qubes with new software (including apps that are not in the official repositories!), and create new templates.
2.1 Activating pre-installed apps
We have a template called debian-12. We would like Salt to create a “vault” qube based on debian-12 that is never connected to the internet, and that we will only use for the app KeepassXC.
Luckily, KeepassXC comes pre-installed in the template debian-12, so we can simply tell Salt to make it available in the app menu. We write our state configuration file /srv/user_salt/vault.sls as follows:
vault--create-qube:
qvm.vm:
- name: vault
- present:
- template: debian-12
- label: black
- prefs:
- label: black
- netvm: none
- features:
- set:
- menu-items: org.keepassxc.KeePassXC.desktop org.gnome.Terminal.desktop
As a result, running sudo qubesctl state.sls vault saltenv=user will make Salt create the vault qube if it’s not there, and make sure that it has KeePassXC in its app menu.
To make things easier, we would like Salt to automatically take care of the vault when we run highstate. We write the following in the top file /srv/user_salt/top.sls:
user:
dom0:
- vault
The command sudo qubesctl state.highstate will now automatically run the “vault” state.
2.2 Installing new apps
We would like to have a “messaging” qube for communicating with our friends through an app called Telegram. However, Telegram is not part of the debian-12 template, so we’ll have to install it.
Luckily, Telegram is available in the official repository. We can therefore tell Salt to create the “messaging” qube and make sure that Telegram is installed in the debian-12 template by writing the state configuration file /srv/user_salt/messaging.sls as follows:
{% if grains['id'] == 'dom0' %}
messaging--create-qube:
qvm.vm:
- name: messaging
- present:
- template: debian-12
- label: yellow
- prefs:
- label: yellow
- features:
- set:
- menu-items: org.telegram.desktop.desktop org.gnome.Nautilus.desktop
{% elif grains['id'] == 'debian-12' %}
messaging--install-apps-in-template:
pkg.installed:
- pkgs:
- telegram-desktop
{% endif %}
There you go! As we need to run the install process in the debian-12 template, we have to target debian-12 when we make Salt execute this state: sudo qubesctl --targets=debian-12 --show-output state.sls messaging saltenv=user.
Note on the "{% ... %}" syntax
This state configuration file has two parts. In the first part, we wrote the instructions that Salt has to execute while running in the admin qube dom0, while the second part is about installing Telegram, which must be executed in the template debian-12. To have everything in the same file but ensure that the right part gets executed in the right qube, we decided to use a Jinja “if statement” to modify the state configuration file depending on what qube Salt is currently running the instructions for.
Similarly as in the previous section, we can have Salt apply this state while targeting both dom0 and debian-12 when running highstate. We add the following to the top file /srv/user_salt/top.sls:
user:
dom0 or debian-12:
- messaging
This makes the command sudo qubesctl --targets=debian-12 --show-output state.highstate automatically create a messaging qube with Telegram as part of its app menu.
2.3 Creating a “non-free” template
We would like to create a “conferencing” qube with the software Skype to communicate with our family. Skype, however, is not available from the official debian-12 repository because it is distributed under a proprietary software licence: we will have to add an external repository to be able to install it.
As Skype is not in the official repository, we consider that there is a non-zero risk that it compromises the security of the template during its installation process. Because we want to trust our default templates, we decide to create a new “nonfree” template to install this proprietary software.
We write our state configuration file /srv/user_salt/conferencing.sls as follows:
{% if grains['id'] == 'dom0' %}
conferencing--create-nonfree-template:
qvm.clone:
- name: nonfree
- source: debian-12
conferencing--create-app-qube:
qvm.vm:
- name: conferencing
- present:
- template: nonfree
- label: yellow
- prefs:
- label: yellow
- features:
- set:
- menu-items: skypeforlinux.desktop org.gnome.Nautilus.desktop
- require:
- qvm: conferencing--create-nonfree-template
{% elif grains['id'] == 'nonfree' %}
conferencing--download-key:
cmd.run:
- name: curl --output /etc/apt/keyrings/skype.asc https://repo.skype.com/data/SKYPE-GPG-KEY
- env:
- https_proxy: http://localhost:8082
- creates:
- /etc/apt/keyrings/skype.asc
conferencing--add-repository:
pkgrepo.managed:
- name: deb [signed-by=/etc/apt/keyrings/skype.gpg] https://repo.skype.com/deb stable main
- file: /etc/apt/sources.list.d/skype-stable.list
- key_url: /etc/apt/keyrings/skype.asc
- aptkey: False
- require:
- cmd: conferencing--download-key
- require_in:
- pkg: conferencing--install-apps
conferencing--install-apps:
pkg.installed:
- pkgs:
- skypeforlinux
{% endif %}
Running this state makes Salt create a “conferencing” app qube based on a new template called “nonfree”, in which Salt makes sure that Skype is installed from its external repository. To run this state, we target our new “nonfree” template with the command:
sudo qubesctl --targets=nonfree --show-output state.sls conferencing saltenv=user
Let’s add this state to the top file, so that it is applied automatically when we run highstate! At the end of the top file /srv/user_salt/top.sls, we write:
user:
dom0 or nonfree:
- conferencing
We can now run highstate with the command:
sudo qubesctl --targets=nonfree --show-output state.highstate
Tip: Here is what our top file would look like if we would include all the states from this guide so far.
user:
dom0:
- salty
- disconnected
- vault
dom0 or debian-12:
- messaging
dom0 or nonfree:
- conferencing
With this top file, we would run highstate with:
sudo qubesctl --targets=debian-12,nonfree --show-output state.highstate
We could also directly target all our templates with:
sudo qubesctl --templates --show-output state.highstate
2.4 Useful links
I tried to be as concise as I could. Please let me know if you have any further questions! Here are some related links.
- Qubes docs on installing software
- Qubes docs on trusting your templates
- Basic Salt syntax, an extremely useful primer from the official Salt documentation
- extrepo-data, a list of external repositories for many apps that are not in the official repositories
In the final part of this guide, we will learn how to make Salt perform automated backups of our qubes with Wyng.
Part 3: Backing up Qubes
One of the biggest advantages of using Salt with Qubes OS is that we would just need to copy and run our state configuration files to completely recreate our system. Saving those files would however not be sufficient for a backup, because that would not contain any our personal data.
We could use the official Qubes Backup tool to create full backups of our qubes, but as of now this tool does not support creating incremental backups, which make the backup process much more performant.
In this part of the guide, we are going to create Salt states that can make fast incremental backups of our qubes.
3.1 Downloading Wyng
Wyng is a backup software written by @tasket that we can use to make incremental backups of the contents of our qubes in a very efficient manner. Its project page gives a detailed explanation of how it works.
It should be noted that a tool called wyng-util-qubes is also currently being developed and makes it very easy to use Wyng with Qubes OS. One of the advantages of using this tool is that, on top of backing up the contents of our qubes with Wyng, this tool also backs up their configuration. At the time of writing, however, wyng-util-qubes does not yet support installations of Qubes OS with a BTRFS partitioning scheme. For this reason, and because the configuration of our qubes is already saved in our Salt state configuration files, we are going to use Wyng directly instead of wyng-util-qubes.
To use Wyng, we first have to copy it to dom0. To do so, can we download and extract the main branch of its repository in a trusted qube called “disp8265” with the command:
curl --location https://github.com/tasket/wyng-backup/archive/refs/heads/main.tar.gz | tar --extract --gzip
Note on the version of Wyng used in this guide.
The above command downloads the latest recommended version of Wyng, which is Wing v0.8beta at the time of writing. It might be that some parts of this guide stop working with a more recent version of Wyng, in which case the guide will have to be updated.
The entire Wyng program is contained in the file wyng-backup-main/src/wyng, which is in the directory that we just created. We can follow the instructions shown on the project page to verify the authenticity of this file. Once we trust this file, we can copy it to dom0 by opening a dom0 terminal and running:
Make sure to understand the risks of copying files to dom0 before executing this command.
qvm-run --pass-io disp8265 'cat wyng-backup-main/src/wyng' > wyng
The file wyng is now in our home directory in dom0. We could mark this file as executable with the command chmod +x wyng and run Wyng directly from the command line, but we won’t: we are going to use Salt! But first, we have to create a BTRFS subvolume.
3.2 Creating a BTRFS subvolume
When we installed Qubes OS on our machine, we decided to use BTRFS instead of the default partition scheme. To be able to use Wyng with BTRFS, the files that contain the filesystems of our qubes must be located inside of a BTRFS subvolume. This is not the case by default: the files that contain the filesystems of our qubes are located under /var/lib/qubes, which is not a BTRFS subvolume but a regular directory.
We use the following commands to create a BTRFS subvolme:
qvm-shutdown --all --wait --force
sudo mv /var/lib/qubes /var/lib/qubes-old
sudo btrfs subvolume create /var/lib/qubes
shopt -s dotglob
sudo mv /var/lib/qubes-old/* /var/lib/qubes
sudo rmdir /var/lib/qubes-old
Tip: There is a longer and riskier method that does not involve moving files across subvolumes.
- Shut down all qubes with
qvm-shutdown --all --wait --force - Rename
/var/lib/qubesto/var/lib/qubes-old - Create a snapshot of the root subvolume at the location
/var/lib/qubes:sudo btrfs subvolume snapshot / /var/lib/qubes - Delete everything in the new subvolume
/var/lib/qubesexcept the directory/var/lib/qubes/var/lib/qubes-old - Move the contents of
/var/lib/qubes/var/lib/qubes-oldto/var/lib/qubes(don’t forget the hidden files) - Delete the now empty directory
/var/lib/qubes/var/lib/qubes-oldand its empty parents
Once this is done, the output of the command sudo btrfs subvolume list / should contain a line that ends with /var/lib/qubes, and our system should function normally.
3.3 Making local backups
We would like to configure Wyng to create encrypted incremental backups that we will save on our machine in a service qube called “sys-backup”. The idea is that we’ll back up our qubes regularly and, from time to time, we’ll copy the backup archive in sys-backup to some external storage, or upload it to a remote server.
We create the following files:
/srv/user_salt/backup/map.jinja
{# This file holds the backup passphrase and general configuration #}
{# The passphrase must not contain any single quote character (') #}
{% set backup = {
'passphrase': 'my-backup-passphrase',
'qubes': ['vault', 'messaging'],
'volumes': ['appvms/vault/private.img', 'appvms/messaging/private.img'],
'source': '/var/lib/qubes',
'destination': '/home/user/qubes.backup',
} %}
/srv/user_salt/backup/init.sls
# Install Wyng and create the service qube "sys-backup" for storing backups
{% from 'backup/map.jinja' import backup %}
backup--install-dependencies:
pkg.installed:
- pkgs:
- python3-pycryptodomex
- python3-zstd
backup--install-wyng:
file.managed:
- names:
- /usr/local/bin/wyng:
- source: salt://backup/files/wyng
- mode: 755
- /etc/wyng/wyng.ini:
- source: salt://backup/files/wyng.ini
- template: jinja
- context: {{ backup|yaml }}
- makedirs: True
- require:
- pkg: backup--install-dependencies
backup--create-service-qube:
qvm.vm:
- name: sys-backup
- present:
- template: debian-12
- label: yellow
- prefs:
- template: debian-12
- label: yellow
- provides-network: True # "service qube" (see Qubes issue #7298)
- features:
- set:
- menu-items: org.gnome.Terminal.desktop org.gnome.Nautilus.desktop
- require:
- file: backup--install-wyng
/srv/user_salt/backup/configure.sls
# Create a backup archive and add the backed-up qubes to its configuration
{% from 'backup/map.jinja' import backup %}
backup.configure--create-archive:
cmd.run:
- name: |
echo '{{ backup.passphrase }}' | wyng arch-init --unattended
- unless:
- qvm-run sys-backup test -d {{ backup.destination }}
backup.configure--add-volumes:
cmd.run:
- name: |
echo '{{ backup.passphrase }}' | wyng add {{ backup.volumes|join(' ') }} --unattended
- require:
- cmd: backup.configure--create-archive
/srv/user_salt/backup/clear-cache.sls
# Remove the directory /home/user/.cache in each of the backed-up qubes
{% from 'backup/map.jinja' import backup %}
backup.clear-cache--shutdown-app-qubes:
qvm.shutdown:
- names: {{ backup.qubes|yaml }}
backup.clear-cache--clear-cache:
qvm.vm:
- names: {{ backup.qubes|yaml }}
- actions:
- run
- shutdown
- run:
- cmd: rm --recursive --force /home/user/.cache
- shutdown: []
- require:
- qvm: backup.clear-cache--shutdown-app-qubes
/srv/user_salt/backup/send.sls
# Create a backup and copy it to the archive in "sys-backup"
{% from 'backup/map.jinja' import backup %}
include:
- backup.configure
- backup.clear-cache
backup.send--send-backup:
cmd.run:
- name: |
echo '{{ backup.passphrase }}' | wyng send --all --unattended
- require:
- cmd: backup.configure--add-volumes
- qvm: backup.clear-cache--clear-cache
backup.send--shutdown-service-qube:
qvm.shutdown:
- name: sys-backup
- require:
- cmd: backup.send--send-backup
/srv/user_salt/backup/receive.sls
# Overwrite current data with the most recent backup from the archive
{% from 'backup/map.jinja' import backup %}
include:
- backup.configure
backup.receive--shutdown-app-qubes:
qvm.shutdown:
- names: {{ backup.qubes|yaml }}
- require:
- cmd: backup.configure--add-volumes
backup.receive--receive-backup:
cmd.run:
- name: |
echo '{{ backup.passphrase }}' | wyng receive --all --unattended
- require:
- qvm: backup.receive--shutdown-app-qubes
backup.receive--shutdown-service-qube:
qvm.shutdown:
- name: sys-backup
- require:
- cmd: backup.receive--receive-backup
/srv/user_salt/backup/files/wyng
This is Wyng’s executable file which we copied to dom0 in section 3.1.
/srv/user_salt/backup/files/wyng.ini
{#- This file configures the default options for Wyng commands -#}
[var-global-default]
local = {{ source }}
dest = qubes://sys-backup{{ destination }}
This gives us a general configuration file map.jinja where we can enter our password and the list of qubes that we want to include in our backups, as well as the following Salt states:
- The initial state
init.slsmakes sure that Wyng and the python libraries that are required for encryption and compression are installed in dom0, and creates the “sys-backup” service qube. We must run this state at least once before using the other states. We can run this state with:sudo qubesctl state.sls backup saltenv=user - The state
configure.slscreates the backup archive in sys-backup if it doesn’t exist, and ensures that all the backed-up qubes specified in the configuration filemap.jinjaare in the archive configuration. This state is automatically included in other states that require it, but we can also run it manually with:sudo qubesctl state.sls backup.configure saltenv=user - The state
clear-cache.slsshuts down all the backed-up qubes to make sure that there have no program running, then starts them one by one and removes their cache directory/home/user/.cachebefore shutting them down once again. Because it is important to clear the cache before making backups, this step is automatically included as part of the backup process when using the statesend.slsto make backups. Nevertheless, if needed we can run this state manually with the command:sudo qubesctl state.sls backup.clear-cache saltenv=user - The state
send.slsfirst applies the stateconfigure.sls, then clears the cache of all backed-up qubes by applying the stateclear-cache.sls, and finally creates a new backup in the archive in “sys-backup”. This is the state that we’ll run the most. We can run it with:sudo qubesctl state.sls backup.send saltenv=user - The state
receive.slsautomatically applies the stateconfigure.sls, then shuts down all the backed-up qubes and overwrites their data with the data contained in the latest backup found in the archive. It can be run with:sudo qubesctl state.sls backup.receive saltenv=user
Tip: For very long operations, we can run Wyng manually to see progress in real-time.
Making backups manually requires performing more actions than simply applying the send.sls state file, but it is useful because it shows the output of Wyng in real time during the backup process. It can be done by running the following command (preceded by sudo qubesctl state.sls backup.configure saltenv=user in case the backup archive is not yet configured):
sudo qubesctl state.sls backup.clear-cache saltenv=user && sudo wyng send --all
It is also possible to restore a backup manually by running the following command after shuting down all the backed-up qubes:
sudo wyng receive --all
3.4 Making remote backups
We would like to configure Wyng to create encrypted incremental backups that we will directly upload to a remote Nextcloud Server. To do so, we will use rclone to mount the cloud storage in a service qube called “sys-backup”.
This method is not recommended for backing up large qubes, because each backup has to be copied to a virtual file system in the “sys-backup” qube before it can be uploaded to the remote. This means that the “sys-backup” qube must be large enough to hold an entire incremental backup, which may require too much disk space.
We create the following files:
/srv/user_salt/backup/map.jinja
{# This file holds the backup passphrase and general configuration #}
{# The passphrase must not contain any single quote character (') #}
{% import 'templates.jinja' as templates %}
{% set backup = {
'passphrase': 'my-backup-passphrase',
'qubes': ['archive', 'personal', 'vault'],
'volumes': ['appvms/archive/private.img', 'appvms/personal/private.img', 'appvms/vault/private.img'],
'source': '/var/lib/qubes',
'directory': 'qubes.backup',
'remote': ':webdav,url="https://cloud.example.com/remote.php/webdav",vendor=nextcloud,user=username,pass=0o6e86yXVit3PLTQqrPW_3msNJWctCnf:',
'domain': 'cloud.example.com',
'timeout': 3600,
} %}
/srv/user_salt/backup/init.sls
# Install Wyng and rclone and create the service qube "sys-backup"
{% from 'backup/map.jinja' import backup %}
{% if grains['id'] == 'dom0' %}
backup--install-dependencies:
pkg.installed:
- pkgs:
- python3-pycryptodomex
- python3-zstd
backup--install-wyng:
file.managed:
- names:
- /usr/local/bin/wyng:
- source: salt://backup/files/wyng
- mode: 755
- /etc/wyng/wyng.ini:
- source: salt://backup/files/wyng.ini
- template: jinja
- context: {{ backup|yaml }}
- makedirs: True
- require:
- pkg: backup--install-dependencies
backup--create-service-qube:
qvm.vm:
- name: sys-backup
- present:
- template: debian-12
- label: yellow
- prefs:
- template: debian-12
- label: yellow
- provides-network: True # "service qube" (see Qubes issue #7298)
- features:
- set:
- menu-items: org.gnome.Terminal.desktop org.gnome.Nautilus.desktop
- require:
- file: backup--install-wyng
backup--set-volume-size:
cmd.run:
- name: qvm-volume extend sys-backup:private 10Gi
- require:
- qvm: backup--create-service-qube
backup--set-firewall-rules:
file.managed:
- name: {{ backup.source }}/appvms/sys-backup/firewall.xml
- source: salt://backup/files/firewall.xml
- template: jinja
- context: {{ backup|yaml }}
- require:
- qvm: backup--create-service-qube
{% elif grains['id'] == 'debian-12' %}
backup--install-rclone:
pkg.installed:
- pkgs:
- rclone
{% endif %}
/srv/user_salt/backup/configure.sls
# Create a backup archive and add the backed-up qubes to its configuration
{% from 'backup/map.jinja' import backup %}
backup.configure--create-mount-point:
cmd.run:
- name: qvm-run sys-backup 'sudo mkdir /mnt/remote && sudo chown user:user /mnt/remote'
backup.configure--mount-remote:
cmd.run:
- name: |
qvm-run --quiet sys-backup rclone mount --rc --vfs-cache-mode=writes '{{ backup.remote }}' /mnt/remote
- bg: True
- require:
- cmd: backup.configure--create-mount-point
backup.configure--wait-mounted:
loop.until_no_eval:
- name: cmd.run
- expected: 'true'
- args:
- qvm-run --pass-io sys-backup 'rclone rc vfs/list | jq ".vfses | length == 1"'
- require:
- cmd: backup.configure--mount-remote
backup.configure--create-archive:
cmd.run:
- name: |
echo '{{ backup.passphrase }}' | wyng arch-init --unattended
- require:
- loop: backup.configure--wait-mounted
- unless:
- qvm-run sys-backup test -d /mnt/remote/{{ backup.directory }}
backup.configure--wait-archive-sync:
loop.until_no_eval:
- name: cmd.run
- expected: 'true'
- timeout: {{ backup.timeout }}
- args:
- qvm-run --pass-io sys-backup 'rclone rc vfs/stats | jq "all(.diskCache | .[\"uploadsInProgress\", \"uploadsQueued\"]; . == 0)"'
- require:
- cmd: backup.configure--create-archive
backup.configure--add-volumes:
cmd.run:
- name: |
echo '{{ backup.passphrase }}' | wyng add {{ backup.volumes|join(' ') }} --unattended
- require:
- loop: backup.configure--wait-archive-sync
backup.configure--wait-volumes-sync:
loop.until_no_eval:
- name: cmd.run
- expected: 'true'
- timeout: {{ backup.timeout }}
- args:
- qvm-run --pass-io sys-backup 'rclone rc vfs/stats | jq "all(.diskCache | .[\"uploadsInProgress\", \"uploadsQueued\"]; . == 0)"'
- require:
- cmd: backup.configure--add-volumes
/srv/user_salt/backup/clear-cache.sls
This is identical to the homonymous state in section 3.3.
/srv/user_salt/backup/send.sls
# Create a backup and upload it to the archive through "sys-backup"
{% from 'backup/map.jinja' import backup %}
include:
- backup.configure
- backup.clear-cache
backup.send--send-backup:
cmd.run:
- name: |
echo '{{ backup.passphrase }}' | wyng send --all --unattended
- require:
- loop: backup.configure--wait-volumes-sync
- qvm: backup.clear-cache--clear-cache
backup.send--wait-backup-sync:
loop.until_no_eval:
- name: cmd.run
- expected: 'true'
- timeout: {{ backup.timeout }}
- args:
- qvm-run --pass-io sys-backup 'rclone rc vfs/stats | jq "all(.diskCache | .[\"uploadsInProgress\", \"uploadsQueued\"]; . == 0)"'
- require:
- cmd: backup.send--send-backup
backup.send--shutdown-service-qube:
qvm.shutdown:
- name: sys-backup
- require:
- loop: backup.send--wait-backup-sync
/srv/user_salt/backup/receive.sls
# Overwrite current data with the most recent backup from the archive
{% from 'backup/map.jinja' import backup %}
include:
- backup.configure
backup.receive--shutdown-app-qubes:
qvm.shutdown:
- names: {{ backup.qubes|yaml }}
- require:
- loop: backup.configure--wait-volumes-sync
backup.receive--receive-backup:
cmd.run:
- name: |
echo '{{ backup.passphrase }}' | wyng receive --all --unattended
- require:
- qvm: backup.receive--shutdown-app-qubes
backup.receive--wait-backup-sync:
loop.until_no_eval:
- name: cmd.run
- expected: 'true'
- timeout: {{ backup.timeout }}
- args:
- qvm-run --pass-io sys-backup 'rclone rc vfs/stats | jq "all(.diskCache | .[\"uploadsInProgress\", \"uploadsQueued\"]; . == 0)"'
- require:
- cmd: backup.receive--receive-backup
backup.receive--shutdown-service-qube:
qvm.shutdown:
- name: sys-backup
- require:
- loop: backup.receive--wait-backup-sync
/srv/user_salt/backup/files/wyng
This is Wyng’s executable file which we copied to dom0 in section 3.1.
/srv/user_salt/backup/files/wyng.ini
{#- This file configures the default options for Wyng commands -#}
[var-global-default]
local = {{ source }}
dest = qubes://sys-backup/mnt/remote/{{ directory }}
/srv/user_salt/backup/files/firewall.xml
{#- This file sets up basic firewall rules for "sys-backup" -#}
<firewall version="2">
<rules>
<rule>
<properties>
<property name="action">accept</property>
<property name="dsthost">{{ domain }}</property>
</properties>
</rule>
<rule>
<properties>
<property name="action">accept</property>
<property name="specialtarget">dns</property>
</properties>
</rule>
<rule>
<properties>
<property name="action">accept</property>
<property name="proto">icmp</property>
</properties>
</rule>
<rule>
<properties>
<property name="action">drop</property>
</properties>
</rule>
</rules>
</firewall>
This gives us a general configuration file map.jinja where we can enter our encryption password as well as the list of qubes that we want to include in our backups, the rclone remote specification as a connection string, and how much time we are willing to wait for each upload operation (in seconds). We also wrote the following Salt states:
- The initial state
init.slsmakes sure that Wyng and the python libraries that are required for encryption and compression are installed in dom0, and installs rclone in the template “debian-12”. It also creates the “sys-backup” service qube, and adds a firewall rule to prevent this qube from conecting to any IP address that does not correspond to the domain of the cloud storage. We must run this state at least once before using the other states. We can run this state with:sudo qubesctl --show-output --targets=debian-12 state.sls backup saltenv=user - The state
configure.slsmounts the cloud storage in “sys-backup”, creates the backup archive if it doesn’t exist, and ensures that all the backed-up qubes specified in the configuration filemap.jinjaare in the archive configuration. This state is automatically included in other states that require it, but we can also run it manually with:sudo qubesctl state.sls backup.configure saltenv=user - The state
clear-cache.slsshuts down all the backed-up qubes to make sure that there have no program running, then starts them one by one and removes their cache directory/home/user/.cachebefore shutting them down once again. Because it is important to clear the cache before making backups, this step is automatically included as part of the backup process when using the statesend.slsto make backups. Nevertheless, if needed we can run this state manually with the command:sudo qubesctl state.sls backup.clear-cache saltenv=user - The state
send.slsfirst applies the stateconfigure.sls, then clears the cache of all backed-up qubes by applying the stateclear-cache.sls, and finally uploads a new backup to the archive through “sys-backup”. This is the state that we’ll run the most. We can run it with:sudo qubesctl state.sls backup.send saltenv=user - The state
receive.slsautomatically applies the stateconfigure.sls, then shuts down all the backed-up qubes and overwrites their data with the data downloaded from the latest backup found in the archive. It can be run with:sudo qubesctl state.sls backup.receive saltenv=user
Tip: For very long operations, we can run Wyng manually to see progress in real-time.
Making backups manually requires performing more actions than simply applying the send.sls state file, but it is useful because it shows the output of Wyng in real time during the backup process. It can be done by running the following command (preceded by sudo qubesctl state.sls backup.configure saltenv=user in case the backup archive is not yet configured):
sudo qubesctl state.sls backup.clear-cache saltenv=user && sudo wyng send --all
We must then run the following command periodically to see how many uploads are still in progress/queued:
qvm-run --pass-io sys-backup rclone rc vfs/stats
The backup is complete when the properties “uploadsQueued” and “uploadsInProgress” returned by this command are both equal to zero.
It is also possible to restore a backup manually by running the following command after shuting down all the backed-up qubes:
sudo wyng receive --all
3.5 Useful links
- Wyng repository, it gives detailed explanations for each command and their arguments
- wyng-util-qubes, a tool that makes it easy to use Wyng with Qubes OS
- docs for rclone, a tool to manage files in the cloud supportting many storage providers
- Jinja templating docs, all the syntax and semantics of the Jinja templating engine
- Salt docs on Jinja, they include descriptions of all the built-in filters such as
regex_replace
It took me quite a while to write this final part of the guide but I’m quite happy with the result. I tried to write it in a way that is easy to adapt to another Qubes install. Of course, feel free to ask if something needs clarification or doesn’t work. Have a nice day.
Appendix: Salt resources made by Qubes users
Guides:
- Qubes configuration management, extensive docs by @gonzalo-bulnes
- @unman’s notes, a good reference on Salt written by a developer of Qubes OS
Formulas:
- Shaker by @unman
- Salt & Pepper, drkhsh’s configuration for automated template deployments
- qubes-salt-examples by @zaz and @kenosen
- Qubes SaltStack configuration of Videos Playback VM (didatic) by @bruno_schroeder
- qusal: Salt Formulas for Qubes OS by @ben-grande
- qubes-for-journalists, a simple journalist-oriented salt repo and wiki by @kenosen
Videos:
