Feedback on trying to use Salt to automated a small task

User Support
1

My goal was pretty simple: I have a debian-10-dev template originally cloned from debian-10, into which I incrementally installed quite a number of packages. I want to automate things, so I don’t have to redo that for debian-11 and further releases. I’m pretty new with Salt, and trying to make sense out of the doc and @unman’s examples.

I started to concentrate on a formula to get packages installed into a template, and I find that disturbingly complicated.

Following examples it is not hard to come up with something like this, let’s put it in /srv/formulas/base/virtual-machines-formula/qvm/debian-dev.sls:

deb-dev-packages.installed:
  pkg.installed:
    - pkgs:
      - autoconf
      - automake
      - build-essential
      - clang
      - g++

From this example I infered a command-line:

qubesctl --skip-dom0 --targets=debian-11-dev state.apply qvm.debian-dev

I must say I’m horrified having to specify --skip-dom0, and when not specifying seeing a silent attempt to install all those packages in dom0, and also no error when making a typo in the VM name passed to --targets.

I tried to understand why I would need a .top file there, as all examples ship one alongside with the every .sls file. I thought they could be used to specify the target qube’s name, but each time I tried qubesctl ignored that it was not dom0 I specified in my .top file. And finally I just removed it, and qubesctl appears to do its job without a problem. Could someone shed some light here ?

From what I gather, I have to write separate formulas for the actions to be executed in dom0 (template cloning, any parameter adjustments) and for the ones to be executed in the new template (package installations, etc). This sounds sensible in a way, but I can’t see how they can be chained in a single invocation, ie. as is there is in fact little value to replace a call to qvm-clone by one to qubesctl with a 3-lines formula, if I also have to run a second qubesctl to install the packages. Surely it must not be that complicated to come up with an example that would in one go clone the template and add packages to the clone ?

I’ve read a bit of SaltStack Challenges and Experiences, and from this small incursion in the world of Salt I think I can understand @restive’s criticisms, and can’t help to wonder if Salt is indeed the right tool for the task: at the very least, the learning curve is quite steep. And if those examples really reflect the best we can do from a user’s perspective, is it really worth it ?

2

From reading, .top files are used to assign states to VMs.
Top Files
So using the information above a starting skeleton of a .top file could be:

environment: base:
target_matching_clause: 'debian-10-dev':
- statefile1 - debian-dev

At the risk of wandering far away from what the intended goal is…
Maybe the kali_template github repo by @unman would be more similar to what the goal is here; modifying a debian template by installing packages.
It might be clearer to change the name of debian-10-dev.sls to simply install.sls, and the .top to just install.top.
For instance the salt tree could be /dvlp_template/dvlp, then install.top would be:

base:

template-dvlp:

- dvlp.install

Apologies if this missed the mark. Maybe @unman could help.

3

My goal was pretty simple: I have a debian-10-dev template originally cloned from debian-10, into which I incrementally installed quite a number of packages. I want to automate things, so I don’t have to redo that for debian-11 and further releases. I’m pretty new with Salt, and trying to make sense out of the doc and @unman’s examples.

I started to concentrate on a formula to get packages installed into a template, and I find that disturbingly complicated.

Following examples it is not hard to come up with something like this, let’s put it in /srv/formulas/base/virtual-machines-formula/qvm/debian-dev.sls:

deb-dev-packages.installed:
  pkg.installed:
    - pkgs:
      - autoconf
      - automake
      - build-essential
      - clang
      - g++

From this example I infered a command-line:

qubesctl --skip-dom0 --targets=debian-11-dev state.apply qvm.debian-dev

It’s covered in both the manpage and in --help - if you dont
specify --skip-dom0 salt will install packages in dom0 as you asked.
If you want to guard against this you can use some defensive measures - see below.

I tried to understand why I would need a .top file there, as all examples ship one alongside with the every .sls file. I thought they could be used to specify the target qube’s name, but each time I tried qubesctl ignored that it was not dom0 I specified in my .top file. And finally I just removed it, and qubesctl appears to do its job without a problem. Could someone shed some light here ?

You might find these notes and examples helpful, as a
basic introduction.
One reason to write separate states is that it’s a key principle in salt
to use a modular approach.

Here’s one approach that you can take:
Put these files in /srv/salt/test.

clone.top:

base:
  dom0:
    - match: nodegroup
    - test.clone

clone.sls:

include:
  - template-debian-11-minimal

qvm-clone-id:
  qvm.clone:
    - require:
      - sls: template-debian-11-minimal 
    - name: template-test
    - source: debian-11-minimal

test.top:

# vim: set syntax=yaml ts=2 sw=2 sts=2 et :

base:
  '*':
  - test.test

test.sls:

# vim: set syntax=yaml ts=2 sw=2 sts=2 et :

include:
  - test.clone

{% if grains['nodename'] != 'dom0' %}

install_pkgs:
  pkg.installed:
    - refresh: True
    - pkgs:
      - p7zip
      - vim

{% endif %}

Then you can call:
qubesctl --show-output --targets=template-test state.apply test.test

You target the template that you are creating in test/clone.sls,
and you apply the state in test.sls
In test.sls you use an include statement, which calls clone.sls,
and that creates the new template.
After the include is a bit of defensive yaml - not pretty but effective.

Oh, and you’ll notice there is another include statement there - it
makes sure that there is a minimal template there to clone.

template-debian-11-minimal.sls -

template-debian-11-minimal:
  pkg.installed:
    - name:   qubes-template-debian-11-minimal
    - fromrepo: qubes-templates-itl

Building your state tree is crucial, and breaking down the states in to
clear steps with informative names is good practice.

Nothing here is hard to understand, I think. It’s basic salt.
If you put together a series of state trees, you can use them to
recreate your whole system with a few calls to qubesctl,or,(if you
have taken a modular approach), set up some part of your system, at
will.

This is over long and is probably riddled with spelling and layout
errors. I hope the intent is clear, if nothing else.

4

Yes the option is documented, but not in a way that would make it clear why such a strange-looking choice was made: why would a formula, which is specified (through recipe text in its .top file) to apply to a given VM, would also by default also apply to dom0 ? In which situations is this concept useful ?

Nice, thanks. Through which channel would you prefer feedback on this ? Github makes it easy to create issues from file contents, would that fit ?

Yes, but then let’s step back and look at this command: we request application of a formula to a specific qube (which does not exist yet), and then behind the scene we end up getting a formula applied to dom0 - and the very formula that we request for our qube is not applied to dom0 only because we introduce some jinja ifdef. That just does not feel right at all.

Some details that feel awkward to me (and how I imagine they could look “better”, for some definition of that word):

  • qvm.clone looks like an imperative statement, as opposed to the declarative nature of pkg.installed and friends.
    One thing that in my mind distinguish Salt from shell scripting is precisely that Salt is declarative and not imperative. If I see a qvm.clone imperative-looking statement, I tend to wonder “but what happens when a qube with that name exists ?”, as typically the imperative approach would allow “error out”, “overwrite” (preferably explicitly), or “ignore” (also preferably explicitly)… while we probably don’t want a formula to error out when it has already done its job; whereas if I see a qvm.cloned declarative statement basically saying “it has to be cloned from that other qube” (which could be tracked eg. using a cloned-from-xxx tag), I would expect the formula to succeed if that is the case (and naturally fail if is not, so that the subsequent steps of a formula do not get applied to an unrelated qube by mistake).
  • we request application of a formula to an unborn target.
    We ought to be able to specify the target state in a fully declarative way, possibly without having to care who’s going to do the job. Something like:
    test-template.cloned:
  qvm.cloned:
    - name: test-template
    - source: whatever

test-template.packages:
  target: test-template
  pkg.installed:
    - ...
    Not sure how such a thing could fit in Salt’s way of doing things, though. I guess there is already some Salt-native idiom to deploy a VM and subsequently work on it ? Or maybe those idioms would tend to be tailored to “handling large numbers or VMs from a given formula”, which would make it more verbose/less suitable for our qubes ?