A bit of help with Salt, please?

OK I’m trying to tinker with salt to learn about it.

Basically I’m going from a Sven-like set of scripts to set up a bunch of minimal templates, to doing the same thing in salt.

So the general situation is: Starting with debian-11-minimal, I do a clone to template A. A just has updates run on it; and then I clone it to make B. I add a bunch of “general” things to B (and configure aliases and the like), then from B I clone C1 and C2, and then a “tree” grows from there.

Cloning turns out to be something that can be run in the dom0 environment, and so I can write a bunch of top files, one for each template, and I can even write them so that they happen in the proper order. And it will all happen with a simple qubesctl state.highstate

But by itself it misses the point; I want to do the configuration of, say B, before cloning C1 and C2 from it. And from what I could see a lot of this configuration has to happen in a different environment; for example the configuration of B must happen in the B environment.

So the top file for the B template, b-setup.top looks something like this:

            - b-clone
            - b-configure

And things happen in the proper order when I enable b-setup.top.

OK, so b-clone is an sls file that does the clone, and, because commands like qvm-prefs, qvm-tags and qvm-features are commands run in dom0 that specify their target vm, I can put those in b-clone.sls as well.

b-configure, on the other hand, oftentimes has to append things to files on template B, or copy files to B. All stuff best done ON the B template, and using the B environment will cause it to happen…and it happens after the cloning. (Apparently the things in a top file execute seriatim.)

The trouble begins when I enable a-setup.top and b-setup.top at the same time and then invoke qubesctl --target b state.highstate. That should update B, working on A if necessary.

What happens is a-clone is run, then b-clone. As b-clone is being run, however, a-configure is also being run. I actually wanted b-clone to wait until a-configure was done.

(And then, if there is a C1 and C2 enabled, they too will be cloned, even though I was trying to generate B and no further. But as far as I can tell, neither C2 nor C2 gets configured. The answer to this one, at least, seems easy; just be careful what you enable.)

So, I’ve been totally unable to figure out how to get b-clone to depend on a-configure in such a way that it won’t start until a-configure is done. There doesn’t seem any provision at all for getting top files to depend on each other. Apparently the system is smart enough to recognize the clones depend on each other, but I can’t get them to depend on something not in the dom0 environment.

How do I do this?

Ultimate goal: Ideally I’d like to be able to readily–1) generate all vms in the tree from nothing but a copy of debian-11-minimal. 2) update a particular VM, including updating its entire line of ancestry. 3) cause updates to everything, for instance when debian-11 has an update, applying the update to every VM in the tree without having to regenerate them. (That’s a nice-to-have; certainly I could blow all the VMs away and regenerate them all with one command…overnight.

if I remember correctly some experiments done on uyuni, You can subdue next task with requisites state:

Requisites are indeed the way for structuring calls between states in
the same minion
, but I do not think they are of use here.
For this sort of structure in salt you would use OverState or
(better) orchestration. That needs salt-run which isn’t handled in the
Qubes context.
Because I am lazy I use separate calls to qubesctl:
qubesctl <clone C1, C2>

I’ve pretty much settled on enabling the top file for A, doing a qubesctl --targets A state.highstate, then disabling that top file and enabling the one for B.

This is tedious, but that seems to be the only way to keep them from interfering with each other. All the documentation I’ve seen, though very helpful with what it DOES supply, just addresses the case of creating and configuring ONE template (or AppVM) at a time. (I will probably write a script to do this triplet of operations, very very soon! It would need target name and top file name stem as the parameters.)

I’ve made a lot of progress on salt, but now I’m noticing a lot of repetitive code. Once I learned I could set a jinja variable at the top of the .sls file, it at least makes it easier to copy the sls files (create to make the thing, configure to install stuff on it–doesn’t apply to named DVMs or even, usually their templates) for one template, to another template. (Make the copy, alter two or three lines at the top, and maybe I need to change a label (color) in the middle of the file, but I could manage that too with a variable at the top.) But doing that forced me to realize that there’s a LOT of commonality between them. So I’m trying to figure out how to get Jinja to simply do a C-style include of some other file into an .sls. (I know that Jinja variables don’t seem to be retained in a salt included file, or I’d do that–salt includes have already cut down some redundancy.)

Another source of repetitiveness is that apparently I can’t do more than one file.managed or file.link in a single labeled state, so if I need to move four files, I need four states. If I need to make soft links to three of them, I need seven states. If I could at least combine the manage and link, I could cut it down some.

If that’s confusing (because, perhaps I’m not using the right nomenclature), what I’m saying is I can’t do this:

          source: <first file, in dom0>
          name: <first file, in template location>
          source: <second file, in dom0>
         name:  <second file, in template location>

but instead I must do this:

          source: <first file, in dom0>
          name: <first file, in template location>
         source: <second file, in dom0>
         name:  <second file, in template location>

(I am typing those from memory whilst on a non-qubes physical machine (poor benighted me), so I probably have some obscure point of syntax wrong.)

Addendum: Done as an additional comment rather than an edit because the person I’m interacting with is an e-mail user:

I think I found how to do a jinja include. (Basically I had to look for a tutorial rather than the standard Jinja documentation, which focuses on other things to the extent it was effectively noise.)

And another:

Multiple files controlled with one state:

manage multiple files in same state · Issue #52130 · saltstack/salt · GitHub

Though apparently links must still be done in another state…or states. (Clearly that’s a case where you want to ensure the links happen AFTER the files are managed.)

Tedious indeed - try:

qubesctl --targets A state.apply STATE_A
qubesctl --targets B state.apply STATE_B

If you are deploying multiple states then templating is the way to go.
If you are offering a series of states for choice, not so much.

Combined -

    - user: root
    - names:
      - /etc/one:
        - source: salt://1
      - /etc/two:
        - source: salt://2
      - /etc/three:
        - source: salt://3
    - order: 1

    - names:
      - /rw/config/one:
        - target: /etc/one
      - /rw/config/two:
        - target: /etc/two
      - /rw/config/three:
        - target: /etc/three
    - order: 2

Note use of - names: and - order:

First off, I’d like to thank you for all your help.

One other thing (for anyone else reading) to note about the combined states is that the names on both the manage and symlinks end in a colon (:slight_smile: (e/g. /etc/one**:** and /rw/config/one**:** You showed that, but based on what I saw while researching this, many people reading it will miss it.

Well, this had a totally unexpected (and undesired) result. Let me give some background. I have A-setup.top, inside of that is dom0: A-create(.sls) and A: A-configure(.sls); like this:

          - A-create
           - A-configure

OK, so trying your command like this: sudo qubesctl --targets A state.apply A-setup does not work. It doesn’t seem to recognize top files. So, maybe, I should do the create and configure each, seriatim (in which case it’s two commands…) So I tried sudo qubesctl --targets A state.apply A-create which looked good at first; it created the A VM. The problem is, it then tried to run A-create again within the A environment and this not only wasted time since it had to start up the manager virtual machine and A itself (that takes at least as long as a clone), it failed because qvm.clone, qvm.prefs, and qvm.features were not found in SLS 'A-create' (Which is funny, because they actually ARE there.)

I didn’t even bother to run the A-configure at this point. Until that’s straightened out and I have a command that will run one .top file (or can find a way to gracefully combine my two sls files), I’m better off writing a simple script to enable top, do a highstate, disable top.

Undesired, yes.
Absolutely to be expected.

  1. Environment has a specific use in salt. It’s a file structure
    containing state and top files.
    The default is base -which points at /srv/salt
    Qubes gives you another environment, user, which you can set up from
    /srv/salt/qubes - that provides /srv/user_salt and /srv/user_pillar
    You can also set up other environments.
    Look in /etc/salt/minion.d/f_defaults.conf
    I don’t.

  2. Top files are used for targeting.

    - A-create
    - A-configure

In the base environment apply A-create to dom0 and A-configure to A.

Qubes has (imo) elegant way of working with, and combining, top files.

  1. state.apply works like this:
    with NO state given, run highstate.
    with state given, run that state against specified targets. (Top file
    not involved.)

  2. qubesctl default is to run states against dom0.
    If you don’t want that you must use --skip-dom0

So the command you used, said “Apply this state to dom0 and A”.
Obviously that doesn’t work,as you discovered.
Either fix up the targeting, or fix the state.

  1. Defensive code in state files.
    You can make sure that states work exactly as you expect with some
    For example, if you want to configure only “A”, put this in the state
{% if grains['nodename'] == 'A' %}
configure stuff....
{% endif %}

Or,for any target except dom0,

{% if grains['nodename'] != 'dom0' %}
configure stuff....
{% endif %}

You could extend this to differentiate different os, or different roles.

Well when I read all of that, I was about to argue with it…then I realized I had misunderstood it, and read it again.

So this is what I did. I’m going to make a slight change to my naming convention; A is a template so I’ll call it A-tmpl

sudo qubesctl state.apply A-tmpl-create

(note: no target specified, just the name of the A-tmpl-create.sls file for A-tmpl. Why? Because I want dom0 to see this…but ONLY dom0. Since it’s on by default, I need specify nothing for targets.


sudo qubesctl --targets A-tmpl --skip-dom0 state.apply A-tmpl-configure

Which ran it in A-tmpl and not in dom0.

(I then went on to run corresponding commands for A-dvmt (the DVM Template based on A), then the named DVM A, itself. In both cases there was no need to run a -configure file. Although some DVM Templates do need to be configured, the one I used is not one of them.)

I still can’t create and configure a VM with one qubesctl command, but it can be scripted and the other way often required three commands in the script instead of two (to enable and disable the top file) to say nothing of having to ensure first that no other top files were enabled by mistake.

Just out of curiosity, I ran the config file on dom0 (as if it were a create file) and it was a mix of errors and…I don’t know what. Probably wasn’t a good idea. I may have installed some software on dom0 by doing so. That’s an argument for writing the guards you suggested.

To do this in one you either need jinja in there, or use top file.
Also,if you do this you will end up with a specific state,which
goes against modular best practice.
What if you want to apply that configuration to another, existing, qube?

Those are important methodological points, of course.

To (possibly) clarify what I’m trying to do here, I basically started out with Sven’s minimal qube generation; this is actually an evolution of that. So far my expectation is that I will generate VMs from nothing, but I do want to be able to update without re-creating so I’ve been keeping that in the back of my mind.

So right now my emphasis is on generating all templates from scratch, but I’ve noted that keeping creation and configuring separate makes it easier to re-apply a state to a VM. So no, I don’t really want to combine them, unless I figure out a way to make it behave like two different files depending on the target–in which case I still might have to call it twice, so no gain. [Note: we name things a bit differently. What you call template and appvm VMs “clones” and “creates” are both “creates” to me; while your “installs” and “configures” are both “configures” to me; the name of the sls file specifies whether it’s for a template or an appvm.] In fact my configure sls files do a salt include on the “ancestor” ones. In other words Template B, cloned from Template A, has a configure file that includes Template A’s configure file. About a week ago I established that I was able to delete ancestor templates, then run salt against a descendant template which had been cloned directly from debian-11-minimal (rather than its immediate ancestor), then had prefs and features set manually. The resulting qube had all software installed, including stuff done on ancestor templates. That proved to me that I could add a package to the ancestor, then add it to its descendants without having to re-create them, by simply running each descendant configure sls. [But I may have broken that, see below.] That’s good for updates, though I’d not do it that way for initial generation.

My creates are so much alike that every one (except one unique case) now just sets a bunch of jinja variables and calls a jinja include. (Jinja variables don’t seem to carry over through a salt include.) In one or two cases, there may be a unique special-case state after the include (an example of this is the 20GB size setting for cacher). So yes, the “main” create.sls is specific to a particular VM, but you gotta give it a name somewhere in the chain and there’s literally no redundant code this way. The includes are different depending on whether I’m creating a template VM, a DVM template, a named DVM, or a “regular” AppVM. (It’s early yet–I’ve only done one AppVM so far, and as I move on into “user” VMs I might not do any more, since I don’t want to stomp on user history in those.)

It’s the configure scripts that tend to be more “unique” though sometimes I spot commonality and pull it out into a salt include. (I may change that to jinja; I did those before I really began to understand what Jinja was all about.) Or to put it another way, a create file is very specific to a specific template as it is. (It’s also true that many DVM templates have no configuring to do at all–user settings, maybe. And named DVMs never do.)

In all cases I write a name once at the top of the file, and reference the variable from then on, even in the names of states (which have to be unique). And even there, there’s a common stem in template names (deb11m-something), but that I’ve put into a salt grain, so when debian 12 minimal is truly ready, I can just change the grain, and generate everything with new names.

I mentioned including ancestor template configure (or install) sls files in the descendant template ones; I realized while writing this that putting those jinja guards in may have broken that (they check for the exact target name rather than just “not dom0”). I should be able to fix that with a minor bit of refactoring, though.

I want to conclude by thanking you for your help!