[Salt Help] How to Abstract Base Template Version in State (.sls) Files?

User Support General
,
1
To mods

This is purely a salt question applied to Qubes, and not specific to Qubes. I asked here because I’m configuring a Qubes system, but feel free to move or close as you see fit.

Currently, when using Salt to create a new template based on another or to manage that template like Fedora 43, I have several mentions to Fedora 43 specifically, and it would be nice if I only had to write the version once instead of multiple times.

Currently, I include a sls that includes the name of the template, which I would also like to generalize, and I clone the template. But I have over twenty templates that I have to manually or batch rename, and it’s error-prone and redundant. I also have to manage the templates themselves, as each has its own .sls and .top file to contend with as well.

Has anyone faced this problem before and found an elegant solution, ideally a .sls I can place in a ‘common’ directory and be done with it, updating that one file to update everything? I hate to be a beggar and chooser, and I’d love to hear any solution that I could improve, but the emphasis here is on elegant. I don’t want to trade one debt for another.

2

I put the base in a pillar - then I can pull that in to any state as
needed. Single point to update and universally available - you can use
this either to clone to new templates, or to run in place upgrades.

I never presume to speak for the Qubes team.
When I comment in the Forum I speak for myself.

3

Thanks, I’ll look into that again. I’m currently not familiar with pillars. Are they like global variables? I’ve tried to start using them before, but I wasn’t able to get a clear picture from the salt docs. Are there examples in your unman/shaker GitHub repo?

4

That’s probably my best bet from what I can see, but do you know of any other solutions?

Please excuse my rant.

I’m not ungrateful for your help, but I am passing from frustration and into anger with Salt. It’s one claim to fame is that it abstracts configuration management to make it better and easier, but I’m increasingly frustrated with the feeling that I only traded one problem for another (worse?) problem. I get that Salt is complex by necessity since abstracting all these systems while keeping Salt’s feature set is bound to have complications, but as soon as you graduate from the most basic of configurations, salt is downright ugly, and in every way the YAML is worse and less readable than code because they’ve basically built a coding language on top of YAML. With Bash or Python, I could just make a variable however I want and import it. With pillar, I need to use an entirely different root structure, make sure to add it to the .top file, and write a novel to retrieve it.

My frustration with Salt is growing at an exponential rate compared to the amount of ‘features’ I use, and I’m almost ready to kiss the mouth of the man who invents something that’s not trading one issue for another of equal (or worse) trouble. The only reason I don’t leave Salt right now is because I’m holding out for the hope that I use it long enough to be good enough to negate the issue (i.e. ‘beware the man with one gun’). Maybe the problem is just me, but I’m still upset that something that’s supposed to make this simple has ended up costing me more time and energy now than every other part of this project combined.

Maybe Ansible is better? The readability still looks terrible, so I’m not too hopeful, but maybe that’s just because I’m used to Salt.

5

I find Salt to be simple to use, and extremely readable. Maybe I’m
blind to the issues (Ha)

You can use salt['pillar.get']('codename:latest') and apply that
across all your templates - that seems pretty straightforward to me, and
a very short story.

I never presume to speak for the Qubes team.
When I comment in the Forum I speak for myself.

6

That’s better than I thought. I was reading that I had to make a file under /srv/pillar and include that file in the .top file under base, and then call it with {{ salt['pillar.get']('varname', 'fallback') }} according to the salt docs on pillar. To me, that’s insanely ugly for something I should just be able to include a file for at the top and have ‘varname’. Something like:

...
include:
 - path.to.variables # (.sls)

{{ variables.fedora }}: # The file and this prefix would probably be templates, i.e. {{ templates.fedora }}
  qvm.template_installed: []
...

Maybe I’m just getting overly frustrated because I can’t make it work the way I want it. You know how it is with programming sometimes, when you have a complex problem you just can’t make work. Maybe I’m just not being flexible enough.

Another example, if I want to clone a template I need to confirm that the source exists or install it (qvm.template_installed). I should just be able to do this:

...
clone-<name-of-template>:
  qvm.template_installed:
    - source-template
  qvm-clone:
    - name: name-of-template
    - source: source-template
...

instead, it fails because I can’t use the same module (qvm.) twice in the same id (‘clone-<name-of-template>’). because of this, I can’t have a file (say /srv/salt/common/require-templates.sls) with a qvm.template_installed for each base template, and just include it and require the id of the specific template I need.

...
include:
  - path.to.sls
...

The above executes all states in the included sls, which is contrary to expectation. It should wait for a state to be called, such as in a require statement.

Lastly, and one of my biggest issues, is that when using qvm.vm to manage/create qubes, you have to use - present: in order to create a missing qube, but if the qube already exists, it will not configure a qube. So, if you want a qube to be present or created and in a certain configuration, you must use duplicate information in present to create it and prefs to modify a created qube.

I’m pretty sure I’ve had other issues, but these are just the ones fresh in my memory. My frustration isn’t so much with the software not working per se, just what I see as not doing its job well by being too different/complicated/verbose/constraining.

7

Can you give a particular example? I tried salt['pillar.get']('fedora:latest'), but that didn’t work.

8

Have you set that pillar item? If not, it wont work.

Set the pillar like this in os_versions/init.sls:

debian:
  stable: "13"
  oldstable: "12"

os_versions/init.top:

base:
  '*':
    - os_versions

Import in to the pillar, and refresh.
Then you access in a state file:

{% set debian_stable = salt['pillar.get']('debian:stable') %}

new_template:
  qvm.clone:
    - name: template-test
    - source: debian-{{ debian_stable }}-minimal

I dont think the process of populating the pillar is onerous, and it
gives you what you want - a single file to update, and all your states
will update accordingly.

Of course, there are other approaches:
You can access arguments by reading them in from source.
stable:

13

testprefs_read.sls:

{% set debian_stable = salt['file.read']('/srv/salt/test/stable') %}
 

new_template:
  qvm.clone:
    - name: template-test
    - source: debian-{{ debian_stable }}-minimal

Or, set in yaml - versions.yaml:

debian_stable: 13

testprefs_yaml.sls:

{% import_yaml  "test/versions.yaml" as args %}
 
new_template:
  qvm.clone:
    - name: template-test
    - source: debian-{{ args['debian_stable'] }}-minimal

It sounds as if you would prefer the single file approach. That’s fine
for you. Again, a single file to update and pull in to your states.
Choose whatever works best for you. You might like to create a
config_directory to hold these files to make a central point of reference
from which you can access data in your state files.

I never presume to speak for the Qubes team. When I comment in the Forum I speak for myself.
9

I missed commenting on some of your other issues.

Do this -

...
clone-<name-of-template>_source:
  qvm.template_installed:
    - source-template

clone-<name-of-template>:
  qvm.clone:
    - name: name-of-template
    - source: source-template

Well that’s how include works
If you include a SLS file everything in it is added to the state run.
You can override this with an exclude statement to avoid some IDs from
being run, but the real answer is to refactor and simplify the file you
are including.
Issues like this almost always point to a need to simplify your SLS
files.

There’s an open issue about this.
The simplest work round is to separate the creation from the
configuration. Not a bad thing to do in any case: it simplifies the
state files and avoids duplications.

I never presume to speak for the Qubes team.
When I comment in the Forum I speak for myself.

10

I personally see creating and configuring the VM as part of one state – if the state is run, the VM should both exist and be configured a certain way. I don’t quite understand what this fixes. Currently, I just run a qvm.vm and put the present, prefs, features, etc. under that one module. I did figure out after thinking about it that it doesn’t really matter what I put under present as the label (and the template, if it’s an appvm), because prefs will change it, and it doesn’t really matter what I put as the class (and the template, if it’s a templatevm) under prefs because present (or the creation of the qube) supersedes it. I still don’t see how it would reduce duplication, because it still is written twice.

I actually found a different way of doing it that I really like, and works well with the above understanding. I just require the state that downloads the underlying template and configures it, then I run - clone: under the qvm.vm module. This looks much cleaner to me. Eg.:

include:
  - fedora-43 #This state just calls qvm.template_installed and runs qvm.vm similar to below, requiring the aforementioned state.

my-template:
  qvm.vm:
     - require:
        - fedora-43
     - clone:
        - source: fedora-43
     - prefs:
...

Yeah, to me the word include implies that it’s just loading the states for me to be able to use, like an import. I understand better now that I was just hung up on terminology. I do heavily appreciate this response though, because it made me realize I was approaching Salt wrong. I treated a state (.sls) file like it was the biggest unit of an idea, when they are supposed to be the smallest unit of a coherent idea. I refactored my configs, and things are much smoother now.

It seemed like a builtin when you mentioned it for some reason. I had misinterpreted.
For right now, I really don’t like the pillar system. I want a simple globally-accessible variable, but I also want to automate some things attached to it in the future, poiisbly automating it entirely. Currently, though, I haven’t even automated updating the templates themselves because I haven’t decided how I want to handle the naming. Appending the Fedora or Debian version seems really tacky for a final result, although I may start doing that until I find a better-looking/smoother way. But until I do, I would basically have to juggle naming and renaming templates (because a working, obsolete template is preferred to none at all if an error occurs, so the old template can only be deleted when the new one works and is assigned to the AppVMs. Basically, these solutions seem like more overhead at this point than hardcoding, especially since my refactoring and new way of setting up templates. Exhibit A: if I’m setting the variable in the pillar, why do I need to set it again in the state file? I do like the potential for running shellcode to get the variable, as that seems like more of what I’m looking for, but I absolutely hate the idea of introducing a shell to this any more than necessary, for obvious reasons. For now I think I’ll just stay with hardcoding and looking aroung for/thinking of a solution more apt for this.

To clarify, I do see the value of pillar, but only for much more complex stuff than picking the latest version of a template and setting like defaults. It just seems to me something that’s got too many parts. I may just be being childish about it because of how much I really despise boilerplate on something this high level. But I’m probably also just being picky and may come around to it after marinating on it for a while.

I would like to say lastly again that I’m very grateful for your help. You’ve given very good answers to my questions, and I may come around to some of this after I have time to think about it. I do greatly appreciate anything I get, especially when you’re taking the time to answer questions that end up having stupid simple answers. And some of it has been very helpful, especially that bit about making .sls states smaller. Thanks.