Qubes Salt Beginner's Guide

Qubes Salt Beginner’s Guide

Part 1: Creating our first qubes

As a beginner, Salt seemed daunting to me at first. It took some effort to learn but it was worth it! I’m writing this guide for beginners who enjoy an hands-on introduction with examples.

1.1 Creating personal state configuration directories

Our journey starts with a file found in the base Salt configuration directory in dom0: /srv/salt/qubes/README.rst (GitHub link). In this file we can read:

qubes.user-dirs

Install and maintain user salt and pillar directories for personal state configurations:

/srv/user_salt
/srv/user_pillar

User defined scripts will not be removed on removal of qubes-mgmt-salt by design nor will they be modified on any updates, other than permissions being enforced.

We can activate qubes.user-dirs to create personal state configuration directories. What is qubes.user-dirs, and how do we activate it? This is what we call a state configuration. It is a configuration file that tells Salt what to do to reach a particular state.

To activate qubes.user-dirs, we can follow the instructions found in its configuration file, /srv/salt/qubes/user-dirs.sls (GitHub link):

qubes.user-dirs
===============

Install and maintain user salt and pillar directories for personal state
configurations:

  Includes a simple locale state file

  User defined scripts will not be removed on removal of qubes-mgmt-salt
  by design nor will they be modified on any updates, other than permissions
  being enforced.

Execute:
--------
  qubesctl state.sls qubes.user-dirs

We run the command sudo qubesctl state.sls qubes.user-dirs. Salt applies the corresponding state, and tells us that some files and directories were created. Among these directories we can find /srv/user_salt/: this is the main directory where we’ll place our own state configuration files.

1.2 The top file and running highstate

Running the state qubes.user-dirs also created the file /srv/user_salt/top.sls. Here is what this file looks like before we modify it (GitHub link):

# vim: set syntax=yaml ts=2 sw=2 sts=2 et :
#
# 1) Intial Setup: sync any modules, etc
# --> qubesctl saltutil.sync_all
#
# 2) Initial Key Import:
# --> qubesctl state.sls salt.gnupg
#
# 3) Highstate will execute all states
# --> qubesctl state.highstate
#
# 4) Highstate test mode only.  Note note all states seem to conform to test
#    mode and may apply state anyway.  Needs more testing to confirm or not!
# --> qubesctl state.highstate test=True

# === User Defined Salt States ================================================
#user:
#  '*':
#    - locale

This file is called the top file.

In the future, when we have many state configuration files, it will become quite tedious to run each state one by one with the command sudo qubesctl state.sls my-custom-state. The top file solves that. If we write in this file how to run each state, we get the ability to run all of them with a single command: sudo qubesctl state.highstate. We call this “running highstate”.

1.3 Targeting qubes

There are three lines that are commented out at the end of the top file /srv/user_salt/top.sls:

user:
  '*':
    - locale

If we were to uncomment those lines and run highstate, Salt would run in all targeted qubes (this is what is meant by the * character) the state locale, for which the state configuration file can either be /srv/user_salt/locale.sls or /srv/user_salt/locale/init.sls.

How do we target a qube? By default, the commands qubesctl state.sls my-custom-state and qubesctl state.highstate only target dom0. To make Salt target additional qubes, we can give their names to the --targets argument:

  • sudo qubesctl --targets=fedora-38 state.sls my-custom-state will run my-custom-state targeting dom0 and fedora-38.
  • sudo qubesctl --skip-dom0 --targets=debian-12,untrusted state.highstate will run highstate targeting the qubes debian-12 and untrusted but not dom0.

1.4. Creating a qube with Salt

We have a template called fedora-38. We would like Salt to create a purple qube named “salty” based on this template. We write the state configuration file /srv/user_salt/salty.sls as follows:

salty--create-qube:
  qvm.vm:
    - name: salty
    - present:
      - template: fedora-38
      - label: purple
    - prefs:
      - label: purple

That’s it! Running sudo qubesctl state.sls salty saltenv=user will make Salt create a purple qube named salty. If salty is already present, Salt will just make sure it’s purple but won’t do anything else.

Note on "saltenv=user"

Note that we always need to add the extra argument saltenv=user to the command sudo qubesctl state.sls my-custom-state when we run individual states from the user directory /srv/user_salt/.

To make things easier, we would like to automatically run this state when we run highstate. We add the following to the top file /srv/user_salt/top.sls:

user:
  dom0:
    - salty

Great! Now, the command sudo qubesctl state.highstate will automatically create salty.

1.5 Creating a disconnected qube

We have a template called debian-11. We would like Salt to create a green qube named “disconnected” based on this template, but that has no web browser and no internet access. We write the state configuration file /srv/user_salt/disconnected.sls as follows:

disconnected--create-qube:
  qvm.vm:
    - name: disconnected
    - present:
      - template: debian-11
      - label: green
    - prefs:
      - label: green
      - netvm: none
    - features:
      - set:
        - menu-items: org.gnome.Terminal.desktop org.gnome.Nautilus.desktop

Perfect! We can now make Salt create this qube with the command sudo qubesctl state.sls disconnected saltenv=user.

To make things easier, we would like to automatically run this state when we run highstate. We add the following lines to the top file /srv/user_salt/top.sls:

user:
  dom0:
    - disconnected

Great! Now, the command sudo qubesctl state.highstate will automatically create our disconnected qube.

Tip: How to make Salt create both "salty" and "disconnected" when we run highstate?

We can write the top file /srv/user_salt/top.sls as follows:

user:
  dom0:
    - salty
    - disconnected

1.6 Useful links

I hope this was clear. Here are some links if you’d like to go further:

The next part of this guide will be about creating new templates and installing packages in them.

Part 2: Apps and templates

In this part we’ll learn how use Salt to make qubes with new software (including apps that are not in the official repositories!), and create new templates.

2.1 Activating pre-installed apps

We have a template called debian-12. We would like Salt to create a “vault” qube based on debian-12 that is never connected to the internet, and that we will only use for the app KeepassXC.

Luckily, KeepassXC comes pre-installed in the template debian-12, so we can simply tell Salt to make it available in the app menu. We write our state configuration file /srv/user_salt/vault.sls as follows:

vault--create-qube:
  qvm.vm:
    - name: vault
    - present:
      - template: debian-12
      - label: black
    - prefs:
      - label: black
      - netvm: none
    - features:
      - set:
        - menu-items: org.keepassxc.KeePassXC.desktop org.gnome.Terminal.desktop

As a result, running sudo qubesctl state.sls vault saltenv=user will make Salt create the vault qube if it’s not there, and make sure that it has KeePassXC in its app menu.

To make things easier, we would like Salt to automatically take care of the vault when we run highstate. We write the following in the top file /srv/user_salt/top.sls:

user:
  dom0:
    - vault

The command sudo qubesctl state.highstate will now automatically run the “vault” state.

2.2 Installing new apps

We would like to have a “messaging” qube for communicating with our friends through an app called Telegram. However, Telegram is not part of the debian-12 template, so we’ll have to install it.

Luckily, Telegram is available in the official repository. We can therefore tell Salt to create the “messaging” qube and make sure that Telegram is installed in the debian-12 template by writing the state configuration file /srv/user_salt/messaging.sls as follows:

{% if grains['id'] == 'dom0' %}

messaging--create-qube:
  qvm.vm:
    - name: messaging
    - present:
      - template: debian-12
      - label: yellow
    - prefs:
      - label: yellow
    - features:
      - set:
        - menu-items: org.telegram.desktop.desktop org.gnome.Nautilus.desktop

{% elif grains['id'] == 'debian-12' %}

messaging--install-apps-in-template:
  pkg.installed:
    - pkgs:
      - telegram-desktop

{% endif %}

There you go! As we need to run the install process in the debian-12 template, we have to target debian-12 when we make Salt execute this state: sudo qubesctl --targets=debian-12 --show-output state.sls messaging saltenv=user.

Note on the "{% ... %}" syntax

This state configuration file has two parts. In the first part, we wrote the instructions that Salt has to execute while running in the admin qube dom0, while the second part is about installing Telegram, which must be executed in the template debian-12. To have everything in the same file but ensure that the right part gets executed in the right qube, we decided to use a Jinja “if statement” to modify the state configuration file depending on what qube Salt is currently running the instructions for.

Similarly as in the previous section, we can have Salt apply this state while targeting both dom0 and debian-12 when running highstate. We add the following to the top file /srv/user_salt/top.sls:

user:
  dom0 or debian-12:
    - messaging

This makes the command sudo qubesctl --targets=debian-12 --show-output state.highstate automatically create a messaging qube with Telegram as part of its app menu.

2.3 Creating a “non-free” template

We would like to create a “conferencing” qube with the software Skype to communicate with our family. Skype, however, is not available from the official debian-12 repository because it is distributed under a proprietary software licence: we will have to add an external repository to be able to install it.

As Skype is not in the official repository, we consider that there is a non-zero risk that it compromises the security of the template during its installation process. Because we want to trust our default templates, we decide to create a new “nonfree” template to install this proprietary software.

We write our state configuration file /srv/user_salt/conferencing.sls as follows:

{% if grains['id'] == 'dom0' %}

conferencing--create-nonfree-template:
  qvm.clone:
    - name: nonfree
    - source: debian-12

conferencing--create-app-qube:
  qvm.vm:
    - name: conferencing
    - present:
      - template: nonfree
      - label: yellow
    - prefs:
      - label: yellow
    - features:
      - set:
        - menu-items: skypeforlinux.desktop org.gnome.Nautilus.desktop
    - require:
      - qvm: conferencing--create-nonfree-template

{% elif grains['id'] == 'nonfree' %}

conferencing--download-key:
  cmd.run:
    - name: curl --output /etc/apt/keyrings/skype.asc https://repo.skype.com/data/SKYPE-GPG-KEY
    - env:
      - https_proxy: http://localhost:8082
    - creates:
      - /etc/apt/keyrings/skype.asc

conferencing--add-repository:
  pkgrepo.managed:
    - name: deb [signed-by=/etc/apt/keyrings/skype.gpg] https://repo.skype.com/deb stable main
    - file: /etc/apt/sources.list.d/skype-stable.list
    - key_url: /etc/apt/keyrings/skype.asc
    - aptkey: False
    - require:
      - cmd: conferencing--download-key
    - require_in:
      - pkg: conferencing--install-apps

conferencing--install-apps:
  pkg.installed:
    - pkgs:
      - skypeforlinux

{% endif %}

Running this state makes Salt create a “conferencing” app qube based on a new template called “nonfree”, in which Salt makes sure that Skype is installed from its external repository. To run this state, we target our new “nonfree” template with the command:

sudo qubesctl --targets=nonfree --show-output state.sls conferencing saltenv=user

Let’s add this state to the top file, so that it is applied automatically when we run highstate! At the end of the top file /srv/user_salt/top.sls, we write:

user:
  dom0 or nonfree:
    - conferencing

We can now run highstate with the command:

sudo qubesctl --targets=nonfree --show-output state.highstate
Tip: Here is what our top file would look like if we would include all the states from this guide so far.
user:
  dom0:
    - salty
    - disconnected
    - vault
  dom0 or debian-12:
    - messaging
  dom0 or nonfree:
    - conferencing

With this top file, we would run highstate with:

sudo qubesctl --targets=debian-12,nonfree --show-output state.highstate

We could also directly target all our templates with:

sudo qubesctl --templates --show-output state.highstate

2.4 Useful links

I tried to be as concise as I could. Please let me know if you have any further questions! Here are some related links.

In the final part of this guide, we will learn how to make Salt perform automated backups of our qubes with Wyng.

Part 3: Backing up Qubes

One of the biggest advantages of using Salt with Qubes OS is that we would just need to copy and run our state configuration files to completely recreate our system. Saving those files would however not be sufficient for a backup, because that would not contain any our personal data.

We could use the official Qubes Backup tool to create full backups of our qubes, but as of now this tool does not support creating incremental backups, which make the backup process much more performant.

In this part of the guide, we are going to create Salt states that can make fast incremental backups of our qubes.

3.1 Downloading Wyng

Wyng is a backup software written by @tasket that we can use to make incremental backups of the contents of our qubes in a very efficient manner. Its project page gives a detailed explanation of how it works.

It should be noted that a tool called wyng-util-qubes is also currently being developed and makes it very easy to use Wyng with Qubes OS. One of the advantages of using this tool is that, on top of backing up the contents of our qubes with Wyng, this tool also backs up their configuration. At the time of writing, however, wyng-util-qubes does not yet support installations of Qubes OS with a BTRFS partitioning scheme. For this reason, and because the configuration of our qubes is already saved in our Salt state configuration files, we are going to use Wyng directly instead of wyng-util-qubes.

To use Wyng, we first have to copy it to dom0. To do so, can we download and extract the main branch of its repository in a trusted qube called “disp8265” with the command:

curl --location https://github.com/tasket/wyng-backup/archive/refs/heads/main.tar.gz | tar --extract --gzip
Note on the version of Wyng used in this guide.

The above command downloads the latest recommended version of Wyng, which is Wing v0.8beta at the time of writing. It might be that some parts of this guide stop working with a more recent version of Wyng, in which case the guide will have to be updated.

The entire Wyng program is contained in the file wyng-backup-main/src/wyng, which is in the directory that we just created. We can follow the instructions shown on the project page to verify the authenticity of this file. Once we trust this file, we can copy it to dom0 by opening a dom0 terminal and running:

Make sure to understand the risks of copying files to dom0 before executing this command.
qvm-run --pass-io disp8265 'cat wyng-backup-main/src/wyng' > wyng

The file wyng is now in our home directory in dom0. We could mark this file as executable with the command chmod +x wyng and run Wyng directly from the command line, but we won’t: we are going to use Salt! But first, we have to create a BTRFS subvolume.

3.2 Creating a BTRFS subvolume

When we installed Qubes OS on our machine, we decided to use BTRFS instead of the default partition scheme. To be able to use Wyng with BTRFS, the files that contain the filesystems of our qubes must be located inside of a BTRFS subvolume. This is not the case by default: the files that contain the filesystems of our qubes are located under /var/lib/qubes, which is not a BTRFS subvolume but a regular directory.

We use the following commands to create a BTRFS subvolme:

qvm-shutdown --all --wait --force
sudo mv /var/lib/qubes /var/lib/qubes-old
sudo btrfs subvolume create /var/lib/qubes
shopt -s dotglob
sudo mv /var/lib/qubes-old/* /var/lib/qubes
sudo rmdir /var/lib/qubes-old
Tip: There is a longer and riskier method that does not involve moving files across subvolumes.
  1. Shut down all qubes with qvm-shutdown --all --wait --force
  2. Rename /var/lib/qubes to /var/lib/qubes-old
  3. Create a snapshot of the root subvolume at the location /var/lib/qubes:
    sudo btrfs subvolume snapshot / /var/lib/qubes
    
  4. Delete everything in the new subvolume /var/lib/qubes except the directory /var/lib/qubes/var/lib/qubes-old
  5. Move the contents of /var/lib/qubes/var/lib/qubes-old to /var/lib/qubes (don’t forget the hidden files)
  6. Delete the now empty directory /var/lib/qubes/var/lib/qubes-old and its empty parents

Once this is done, the output of the command sudo btrfs subvolume list / should contain a line that ends with /var/lib/qubes, and our system should function normally.

3.3 Making local backups

We would like to configure Wyng to create encrypted incremental backups that we will save on our machine in a service qube called “sys-backup”. The idea is that we’ll back up our qubes regularly and, from time to time, we’ll copy the backup archive in sys-backup to some external storage, or upload it to a remote server.

We create the following files:

/srv/user_salt/backup/map.jinja
{# This file holds the backup passphrase and general configuration #}

{# The passphrase must not contain any single quote character (') #}

{% set backup = {
    'passphrase': 'my-backup-passphrase',
    'qubes': ['vault', 'messaging'],
    'volumes': ['appvms/vault/private.img', 'appvms/messaging/private.img'],
    'source': '/var/lib/qubes',
    'destination': '/home/user/qubes.backup',
} %}
/srv/user_salt/backup/init.sls
# Install Wyng and create the service qube "sys-backup" for storing backups

{% from 'backup/map.jinja' import backup %}

backup--install-dependencies:
  pkg.installed:
    - pkgs:
      - python3-pycryptodomex
      - python3-zstd

backup--install-wyng:
  file.managed:
    - names:
      - /usr/local/bin/wyng:
        - source: salt://backup/files/wyng
        - mode: 755
      - /etc/wyng/wyng.ini:
        - source: salt://backup/files/wyng.ini
        - template: jinja
        - context: {{ backup|yaml }}
        - makedirs: True
    - require:
      - pkg: backup--install-dependencies

backup--create-service-qube:
  qvm.vm:
    - name: sys-backup
    - present:
      - template: debian-12
      - label: yellow
    - prefs:
      - template: debian-12
      - label: yellow
      - provides-network: True  # "service qube" (see Qubes issue #7298)
    - features:
      - set:
        - menu-items: org.gnome.Terminal.desktop org.gnome.Nautilus.desktop
    - require:
      - file: backup--install-wyng
/srv/user_salt/backup/configure.sls
# Create a backup archive and add the backed-up qubes to its configuration

{% from 'backup/map.jinja' import backup %}

backup.configure--create-archive:
  cmd.run:
    - name: |
        echo '{{ backup.passphrase }}' | wyng arch-init --unattended
    - unless:
      - qvm-run sys-backup test -d {{ backup.destination }}

backup.configure--add-volumes:
  cmd.run:
    - name: |
        echo '{{ backup.passphrase }}' | wyng add {{ backup.volumes|join(' ') }} --unattended
    - require:
      - cmd: backup.configure--create-archive
/srv/user_salt/backup/clear-cache.sls
# Remove the directory /home/user/.cache in each of the backed-up qubes

{% from 'backup/map.jinja' import backup %}

backup.clear-cache--shutdown-app-qubes:
  qvm.shutdown:
    - names: {{ backup.qubes|yaml }}

backup.clear-cache--clear-cache:
  qvm.vm:
    - names: {{ backup.qubes|yaml }}
    - actions:
      - run
      - shutdown
    - run:
      - cmd: rm --recursive --force /home/user/.cache
    - shutdown: []
    - require:
      - qvm: backup.clear-cache--shutdown-app-qubes
/srv/user_salt/backup/send.sls
# Create a backup and copy it to the archive in "sys-backup"

{% from 'backup/map.jinja' import backup %}

include:
  - backup.configure
  - backup.clear-cache

backup.send--send-backup:
  cmd.run:
    - name: |
        echo '{{ backup.passphrase }}' | wyng send --all --unattended
    - require:
      - cmd: backup.configure--add-volumes
      - qvm: backup.clear-cache--clear-cache

backup.send--shutdown-service-qube:
  qvm.shutdown:
    - name: sys-backup
    - require:
      - cmd: backup.send--send-backup
/srv/user_salt/backup/receive.sls
# Overwrite current data with the most recent backup from the archive

{% from 'backup/map.jinja' import backup %}

include:
  - backup.configure

backup.receive--shutdown-app-qubes:
  qvm.shutdown:
    - names: {{ backup.qubes|yaml }}
    - require:
      - cmd: backup.configure--add-volumes

backup.receive--receive-backup:
  cmd.run:
    - name: |
        echo '{{ backup.passphrase }}' | wyng receive --all --unattended
    - require:
      - qvm: backup.receive--shutdown-app-qubes

backup.receive--shutdown-service-qube:
  qvm.shutdown:
    - name: sys-backup
    - require:
      - cmd: backup.receive--receive-backup
/srv/user_salt/backup/files/wyng

This is Wyng’s executable file which we copied to dom0 in section 3.1.

/srv/user_salt/backup/files/wyng.ini
{#- This file configures the default options for Wyng commands -#}

[var-global-default]
local = {{ source }}
dest = qubes://sys-backup{{ destination }}

This gives us a general configuration file map.jinja where we can enter our password and the list of qubes that we want to include in our backups, as well as the following Salt states:

  • The initial state init.sls makes sure that Wyng and the python libraries that are required for encryption and compression are installed in dom0, and creates the “sys-backup” service qube. We must run this state at least once before using the other states. We can run this state with:
    sudo qubesctl state.sls backup saltenv=user
    
  • The state configure.sls creates the backup archive in sys-backup if it doesn’t exist, and ensures that all the backed-up qubes specified in the configuration file map.jinja are in the archive configuration. This state is automatically included in other states that require it, but we can also run it manually with:
    sudo qubesctl state.sls backup.configure saltenv=user
    
  • The state clear-cache.sls shuts down all the backed-up qubes to make sure that there have no program running, then starts them one by one and removes their cache directory /home/user/.cache before shutting them down once again. Because it is important to clear the cache before making backups, this step is automatically included as part of the backup process when using the state send.sls to make backups. Nevertheless, if needed we can run this state manually with the command:
    sudo qubesctl state.sls backup.clear-cache saltenv=user
    
  • The state send.sls first applies the state configure.sls, then clears the cache of all backed-up qubes by applying the state clear-cache.sls, and finally creates a new backup in the archive in “sys-backup”. This is the state that we’ll run the most. We can run it with:
    sudo qubesctl state.sls backup.send saltenv=user
    
  • The state receive.sls automatically applies the state configure.sls, then shuts down all the backed-up qubes and overwrites their data with the data contained in the latest backup found in the archive. It can be run with:
    sudo qubesctl state.sls backup.receive saltenv=user
    
Tip: For very long operations, we can run Wyng manually to see progress in real-time.

Making backups manually requires performing more actions than simply applying the send.sls state file, but it is useful because it shows the output of Wyng in real time during the backup process. It can be done by running the following command (preceded by sudo qubesctl state.sls backup.configure saltenv=user in case the backup archive is not yet configured):

sudo qubesctl state.sls backup.clear-cache saltenv=user && sudo wyng send --all

It is also possible to restore a backup manually by running the following command after shuting down all the backed-up qubes:

sudo wyng receive --all

3.4 Making remote backups

We would like to configure Wyng to create encrypted incremental backups that we will directly upload to a remote Nextcloud Server. To do so, we will use rclone to mount the cloud storage in a service qube called “sys-backup”.

This method is not recommended for backing up large qubes, because each backup has to be copied to a virtual file system in the “sys-backup” qube before it can be uploaded to the remote. This means that the “sys-backup” qube must be large enough to hold an entire incremental backup, which may require too much disk space.

We create the following files:

/srv/user_salt/backup/map.jinja
{# This file holds the backup passphrase and general configuration #}

{# The passphrase must not contain any single quote character (') #}

{% import 'templates.jinja' as templates %}

{% set backup = {
    'passphrase': 'my-backup-passphrase',
    'qubes': ['archive', 'personal', 'vault'],
    'volumes': ['appvms/archive/private.img', 'appvms/personal/private.img', 'appvms/vault/private.img'],
    'source': '/var/lib/qubes',
    'directory': 'qubes.backup',
    'remote': ':webdav,url="https://cloud.example.com/remote.php/webdav",vendor=nextcloud,user=username,pass=0o6e86yXVit3PLTQqrPW_3msNJWctCnf:',
    'domain': 'cloud.example.com',
    'timeout': 3600,
} %}
/srv/user_salt/backup/init.sls
# Install Wyng and rclone and create the service qube "sys-backup"

{% from 'backup/map.jinja' import backup %}

{% if grains['id'] == 'dom0' %}

backup--install-dependencies:
  pkg.installed:
    - pkgs:
      - python3-pycryptodomex
      - python3-zstd

backup--install-wyng:
  file.managed:
    - names:
      - /usr/local/bin/wyng:
        - source: salt://backup/files/wyng
        - mode: 755
      - /etc/wyng/wyng.ini:
        - source: salt://backup/files/wyng.ini
        - template: jinja
        - context: {{ backup|yaml }}
        - makedirs: True
    - require:
      - pkg: backup--install-dependencies

backup--create-service-qube:
  qvm.vm:
    - name: sys-backup
    - present:
      - template: debian-12
      - label: yellow
    - prefs:
      - template: debian-12
      - label: yellow
      - provides-network: True  # "service qube" (see Qubes issue #7298)
    - features:
      - set:
        - menu-items: org.gnome.Terminal.desktop org.gnome.Nautilus.desktop
    - require:
      - file: backup--install-wyng

backup--set-volume-size:
  cmd.run:
    - name: qvm-volume extend sys-backup:private 10Gi
    - require:
      - qvm: backup--create-service-qube

backup--set-firewall-rules:
  file.managed:
    - name: {{ backup.source }}/appvms/sys-backup/firewall.xml
    - source: salt://backup/files/firewall.xml
    - template: jinja
    - context: {{ backup|yaml }}
    - require:
      - qvm: backup--create-service-qube

{% elif grains['id'] == 'debian-12' %}

backup--install-rclone:
  pkg.installed:
    - pkgs:
      - rclone

{% endif %}
/srv/user_salt/backup/configure.sls
# Create a backup archive and add the backed-up qubes to its configuration

{% from 'backup/map.jinja' import backup %}

backup.configure--create-mount-point:
  cmd.run:
    - name: qvm-run sys-backup 'sudo mkdir /mnt/remote && sudo chown user:user /mnt/remote'

backup.configure--mount-remote:
  cmd.run:
    - name: |
        qvm-run --quiet sys-backup rclone mount --rc --vfs-cache-mode=writes '{{ backup.remote }}' /mnt/remote
    - bg: True
    - require:
      - cmd: backup.configure--create-mount-point

backup.configure--wait-mounted:
  loop.until_no_eval:
    - name: cmd.run
    - expected: 'true'
    - args:
      - qvm-run --pass-io sys-backup 'rclone rc vfs/list | jq ".vfses | length == 1"'
    - require:
      - cmd: backup.configure--mount-remote

backup.configure--create-archive:
  cmd.run:
    - name: |
        echo '{{ backup.passphrase }}' | wyng arch-init --unattended
    - require:
      - loop: backup.configure--wait-mounted
    - unless:
      - qvm-run sys-backup test -d /mnt/remote/{{ backup.directory }}

backup.configure--wait-archive-sync:
  loop.until_no_eval:
    - name: cmd.run
    - expected: 'true'
    - timeout: {{ backup.timeout }}
    - args:
      - qvm-run --pass-io sys-backup 'rclone rc vfs/stats | jq "all(.diskCache | .[\"uploadsInProgress\", \"uploadsQueued\"]; . == 0)"'
    - require:
      - cmd: backup.configure--create-archive

backup.configure--add-volumes:
  cmd.run:
    - name: |
        echo '{{ backup.passphrase }}' | wyng add {{ backup.volumes|join(' ') }} --unattended
    - require:
      - loop: backup.configure--wait-archive-sync

backup.configure--wait-volumes-sync:
  loop.until_no_eval:
    - name: cmd.run
    - expected: 'true'
    - timeout: {{ backup.timeout }}
    - args:
      - qvm-run --pass-io sys-backup 'rclone rc vfs/stats | jq "all(.diskCache | .[\"uploadsInProgress\", \"uploadsQueued\"]; . == 0)"'
    - require:
      - cmd: backup.configure--add-volumes
/srv/user_salt/backup/clear-cache.sls

This is identical to the homonymous state in section 3.3.

/srv/user_salt/backup/send.sls
# Create a backup and upload it to the archive through "sys-backup"

{% from 'backup/map.jinja' import backup %}

include:
  - backup.configure
  - backup.clear-cache

backup.send--send-backup:
  cmd.run:
    - name: |
        echo '{{ backup.passphrase }}' | wyng send --all --unattended
    - require:
      - loop: backup.configure--wait-volumes-sync
      - qvm: backup.clear-cache--clear-cache

backup.send--wait-backup-sync:
  loop.until_no_eval:
    - name: cmd.run
    - expected: 'true'
    - timeout: {{ backup.timeout }}
    - args:
      - qvm-run --pass-io sys-backup 'rclone rc vfs/stats | jq "all(.diskCache | .[\"uploadsInProgress\", \"uploadsQueued\"]; . == 0)"'
    - require:
      - cmd: backup.send--send-backup

backup.send--shutdown-service-qube:
  qvm.shutdown:
    - name: sys-backup
    - require:
      - loop: backup.send--wait-backup-sync
/srv/user_salt/backup/receive.sls
# Overwrite current data with the most recent backup from the archive

{% from 'backup/map.jinja' import backup %}

include:
  - backup.configure

backup.receive--shutdown-app-qubes:
  qvm.shutdown:
    - names: {{ backup.qubes|yaml }}
    - require:
      - loop: backup.configure--wait-volumes-sync

backup.receive--receive-backup:
  cmd.run:
    - name: |
        echo '{{ backup.passphrase }}' | wyng receive --all --unattended
    - require:
      - qvm: backup.receive--shutdown-app-qubes

backup.receive--wait-backup-sync:
  loop.until_no_eval:
    - name: cmd.run
    - expected: 'true'
    - timeout: {{ backup.timeout }}
    - args:
      - qvm-run --pass-io sys-backup 'rclone rc vfs/stats | jq "all(.diskCache | .[\"uploadsInProgress\", \"uploadsQueued\"]; . == 0)"'
    - require:
      - cmd: backup.receive--receive-backup

backup.receive--shutdown-service-qube:
  qvm.shutdown:
    - name: sys-backup
    - require:
      - loop: backup.receive--wait-backup-sync
/srv/user_salt/backup/files/wyng

This is Wyng’s executable file which we copied to dom0 in section 3.1.

/srv/user_salt/backup/files/wyng.ini
{#- This file configures the default options for Wyng commands -#}

[var-global-default]
local = {{ source }}
dest = qubes://sys-backup/mnt/remote/{{ directory }}
/srv/user_salt/backup/files/firewall.xml
{#- This file sets up basic firewall rules for "sys-backup" -#}

<firewall version="2">
  <rules>
    <rule>
      <properties>
        <property name="action">accept</property>
        <property name="dsthost">{{ domain }}</property>
      </properties>
    </rule>
    <rule>
      <properties>
        <property name="action">accept</property>
        <property name="specialtarget">dns</property>
      </properties>
    </rule>
    <rule>
      <properties>
        <property name="action">accept</property>
        <property name="proto">icmp</property>
      </properties>
    </rule>
    <rule>
      <properties>
        <property name="action">drop</property>
      </properties>
    </rule>
  </rules>
</firewall>

This gives us a general configuration file map.jinja where we can enter our encryption password as well as the list of qubes that we want to include in our backups, the rclone remote specification as a connection string, and how much time we are willing to wait for each upload operation (in seconds). We also wrote the following Salt states:

  • The initial state init.sls makes sure that Wyng and the python libraries that are required for encryption and compression are installed in dom0, and installs rclone in the template “debian-12”. It also creates the “sys-backup” service qube, and adds a firewall rule to prevent this qube from conecting to any IP address that does not correspond to the domain of the cloud storage. We must run this state at least once before using the other states. We can run this state with:
    sudo qubesctl --show-output --targets=debian-12 state.sls backup saltenv=user
    
  • The state configure.sls mounts the cloud storage in “sys-backup”, creates the backup archive if it doesn’t exist, and ensures that all the backed-up qubes specified in the configuration file map.jinja are in the archive configuration. This state is automatically included in other states that require it, but we can also run it manually with:
    sudo qubesctl state.sls backup.configure saltenv=user
    
  • The state clear-cache.sls shuts down all the backed-up qubes to make sure that there have no program running, then starts them one by one and removes their cache directory /home/user/.cache before shutting them down once again. Because it is important to clear the cache before making backups, this step is automatically included as part of the backup process when using the state send.sls to make backups. Nevertheless, if needed we can run this state manually with the command:
    sudo qubesctl state.sls backup.clear-cache saltenv=user
    
  • The state send.sls first applies the state configure.sls, then clears the cache of all backed-up qubes by applying the state clear-cache.sls, and finally uploads a new backup to the archive through “sys-backup”. This is the state that we’ll run the most. We can run it with:
    sudo qubesctl state.sls backup.send saltenv=user
    
  • The state receive.sls automatically applies the state configure.sls, then shuts down all the backed-up qubes and overwrites their data with the data downloaded from the latest backup found in the archive. It can be run with:
    sudo qubesctl state.sls backup.receive saltenv=user
    
Tip: For very long operations, we can run Wyng manually to see progress in real-time.

Making backups manually requires performing more actions than simply applying the send.sls state file, but it is useful because it shows the output of Wyng in real time during the backup process. It can be done by running the following command (preceded by sudo qubesctl state.sls backup.configure saltenv=user in case the backup archive is not yet configured):

sudo qubesctl state.sls backup.clear-cache saltenv=user && sudo wyng send --all

We must then run the following command periodically to see how many uploads are still in progress/queued:

qvm-run --pass-io sys-backup rclone rc vfs/stats

The backup is complete when the properties “uploadsQueued” and “uploadsInProgress” returned by this command are both equal to zero.

It is also possible to restore a backup manually by running the following command after shuting down all the backed-up qubes:

sudo wyng receive --all

3.5 Useful links

  • Wyng repository, it gives detailed explanations for each command and their arguments
  • wyng-util-qubes, a tool that makes it easy to use Wyng with Qubes OS
  • docs for rclone, a tool to manage files in the cloud supportting many storage providers
  • Jinja templating docs, all the syntax and semantics of the Jinja templating engine
  • Salt docs on Jinja, they include descriptions of all the built-in filters such as regex_replace

It took me quite a while to write this final part of the guide but I’m quite happy with the result. I tried to write it in a way that is easy to adapt to another Qubes install. Of course, feel free to ask if something needs clarification or doesn’t work. Have a nice day.

Appendix: Salt resources made by Qubes users

42 Likes

I just published Part 2: Apps and templates. :grinning:

7 Likes

This is a very good guide, thank you very much

1 Like

Thank you, very good. A good example for SaltStack configuration is this:
https://git.drkhsh.at/salt-n-pepper/file/README.md.html
It is not mine, I just found it and use it.

1 Like

There’s so much to learn… :dizzy_face:
I’m adding this to my todo list for the next days, thanks a lot! :slight_smile:

1 Like

I finally wrote Part 3: Backing up Qubes. :smile:

4 Likes

Thanks! Although I have a long familiarity with Qubes, I never really learned the salt parts… this helps a lot in that regard.

For the Wyng part, I suggest linking to the main project URL instead of tags since ‘main’ features the most recommended version. Currently the new v0.8beta is recommended as generally better just on account of the extra internal checks, making it a more solid choice than v0.3. The new version is also much easier to use.

I would also suggest dropping the link to the announcement for the older version, as it starts out with examples that are a bit out of date

This part sounds a little dicey…

A malicious file or qube could compromise your system through this command!

…so I added my gpg pubkey and signature files to the project, along with verification instructions. That should help facilitate wording in this guide that sounds more sure-footed; the question then shifts to whether you trust my public key, and doing the verify probably in dom0 after unzipping in the dispvm.

Creating a subvolume (3.2): These instructions seem shorter and don’t involve removing files:

cd /var/lib
qvm-shutdown --all --force --wait
sudo mv qubes qubes-old
sudo btrfs subvolume create qubes
sudo mv qubes-old/* qubes

The above should work fine: The mv command will see the src and dest as being the same device because of the way Linux VFS handles nested subvolumes.

Last suggestion: It should mention that Wyng directly deals with only volumes themselves, not qube definitions or VM settings, so only the VM content is backed up this way. A wrapper called wyng-util-qubes exists which performs send/receive functions and makes sure the Qubes settings are included in the process; it also includes the right combination of volumes for each qube and you only need to know the qube names not any special volume names. Much simpler! The only problem is it doesn’t yet handle Btrfs volume paths, so right now it only works for Thin LVM storage.

And thanks again!

3 Likes

Thanks a lot for creating Wyng, and thanks for taking the time to reply!

I added your improvements to the guide.

I have a question: if there is would be a breaking change in the future, some parts of the guide might stop working for readers who are downloading newer versions of Wyng from the main branch. Wouldn’t it be safer if the guide would recommend downloading a specific tagged version instead? Another possibility would be to add a disclaimer such as “this guide was written for Wyng v0.8beta, it might not work with more recent versions”…

Thanks again.

Edit: I added the line shopt -s dotglob to your instructions for creating the BTRFS subvolume, because otherwise the file /var/lib/qubes/.qubes-exclude-block-devices would not be included in the transfer on my machine.

1 Like

I would recommend this instead of instructing to install a specific version of the package because:

  • it is likely better for folks to use new versions of the package as they come out (realistically, some folks won’t notice the version they installed may be outdated, or date changing the version of the package if it is explicitly specified in the instructions)
  • the guide may well be working fine with later versions for a while
  • if the guide stops working with a future version of the package, then the disclaimer offers a good prompt for folks to ask for the instructions to be updated (a good reminder for you or whoever is able to update the guide in the future)
2 Likes

Thanks for you advice. I added a disclaimer in that section.

1 Like

Is it really necessary to download the key in dvm and copy it to dom0? I think better idea will be using proxy inside the template somehow, as we are doing it manually.

1 Like

I think that’s a great idea, using a proxy should work but it is not yet documented in the official Salt docs. It was however already proposed in a GitHub issue by marmarek so should be a safe thing to do.

Maybe adding something like this would be sufficient:

conferencing--set-proxy:
  environ.setenv:
    - value:
        http_proxy: http://127.0.0.1:8082
        https_proxy: http://127.0.0.1:8082

Edit: It isn’t.

Don’t know how much actual it is but:

1 Like
[ERROR   ] State 'qvm.clone' was not found in SLS 'debian-google-template'
Reason: 'qvm.clone' is not available.

local:
----------
          ID: clone-template
    Function: qvm.clone
        Name: debian-12-google
      Result: False
     Comment: State 'qvm.clone' was not found in SLS 'debian-google-template'
              Reason: 'qvm.clone' is not available.
     Changes:

I got such error when try to use gvm.clone in user saltenv… Anyone have an idea what is wrong?

I think this type of error can happen when you try to use qvm commands while targeting other qubes than dom0.

Did you try to apply your Salt state with the following command?

sudo qubesctl --targets=debian-12 --show-output state.sls debian-google-template saltenv=user

If so, this tells Salt to target both debian-12 and dom0 to execute the instructions found in the state configuration file. If the file contains qvm commands, this will not work because qvm commands are not available in debian-12, they are only available in dom0.

The following should work:

sudo qubesctl state.sls debian-google-template saltenv=user

Still the same.

Can you share your state configuration file and the command you use to apply the state?

I must broke something in my salt since recipes in regular salt folder also stop working. Just reinstalled mgmt-salt packages and everything works again. I also wrote recipe that downloading keys and adding external repo today but I will share it after some polishing.

I updated the guide to download the key directly in the template. Thanks a lot for your contribution!

1 Like

One more thing I still can’t figure out. How to download a key from keyserver using proxy. I remember that there was some bug in proxy implementation in gpg while ago.