For those unfamiliar SaltStack is too used by cloud providers to automate the deployment of servers. It just so happens that Qubes is very much like a cloud, so the Qubes Team integrated saltstack onto the system so some qubes could be setup automatically (this is actually the way the default qubes personal, work, etc. were created when you first started Qubes OS).
But the use of SaltStack goes beyond this, it is extremely useful for organizations as it allows them to deploy configurations remotely on multiple computers (related discussion) and some users have been automating their setups this way. You can read more on the docs:
Learning, as in truly understanding, SaltStack is probably very time consuming, but one can just follow unman’s examples for most basic stuff. But in general, I totally agree. This is probably not for the average Qubes user.
I’m saying it’s a time commitment relative to something like terraform, docker compose, vagrant files, bash scripts. I have read through his shaker, and also through the Salt documentation, @gonzalo-bulnes 's split ssh and gpg, the built in salt, the salt for 4.1 (audio vms and gui), the secure drop salt formulas. I understand how to write what they have written, my problem is that Salt provides very little value over a bash script and is difficult to make fit into the use cases that will be common within Qubes.
For example take a look at Unman’s cacher which requires you to run 3 separate commands after installing it. Then take a look at his kali template which depends on cacher but you will have to manually install cache before you can install the template. Which will work assuming no other part of my setup conflicts with his, there is no way to even prefix the resources created.
One of the reasons I use terraform over shell scripts is because I can rebuild my entire stack from the terraform code written and the reason I know I can do that is because if I remove part of the terraform code and hit apply, terraform will update the state of the system to match the configuration that I have defined. With Salt if I remove something and hit high state it does nothing, if its a package I have to add it to pkg.removed, I mean… yikes.
Another bizarre thing is there doesn’t seem to be a graceful way of handling dependencies, you can use an orchestrator but you would essentially be calling salt from inside of salt to do something that seems wayyy too basic to be this hacky.
Then how about parameterisation, how do we do that? Well the only way to do that seems to be pillars which are shipped as part of the formula and make salt code much more complicated to read, take a look at this split-gpg which uses pillars to define the backends. I’m really impressed that he went to this level of effort to make sure that people could have their custom configurations but the trade off here is that the system becomes more difficult to verify and maintain.
I think it’s also a bad sign that we need to import all this stuff as rpms into Dom0, like surely there should be a way to give access to a script to create vms without giving it access to everything else?
I should say that I think the primary reason behind a lot of SaltStacks design choices is probably to do with the speed of execution, which is also part of why the commands are typically sent from master to minion and the minion decides whether or not to execute the command rather than the master having to process that information for every minion.
That, the fact that the expectation is that all the minions will exist at execution time and the other issues I’ve brought up really make it pretty clear that this system doesn’t really gel well with Qubes.
A lot of interesting thoughts and links! Thank you @restive@deeplow
I am not very familiar with Vagrant, but I look forward to see how the conversation evolves. (For what is worth: I’m learning how to use Salt with Qubes OS because that’s what’s in use now, I don’t know it well enough to have strong feelings for or against Salt itself, even in the context of Qubes OS.)
In the meantime:
I am currently tightening execution order in some of my (still unpublished) states using Salt requisites. There are also ways to define dependencies the other way that I haven’t looked much into yet.
It’s early days (for me) using require, and so far I’d say it reads “okay”. At the moment, my main concern is that I haven’t figured out a satisfactory rule to split up formulas. By that I mean: Salt encourages very specialized (granular) states/formulas which are highly reusable, I’d like my formulas to be as self-contained as possible. Salt requires globally unique state IDs, which is consistent with the goal of defining each thing only once, but gets in the way of the latter. I am still thinking about that. While I know not to fight the tools I use, I am still trying to figure out a balance in the line between “ready-to-use-with-some-redundant-definitions” and “nice-and-DRY-but-requiring-some-assembly” (and understanding how all the pieces can be combined).
As far as I can tell, yes, Salt pillar is the way.
(I like that @restive has been identifying trade-offs, I’ll try to do the same!) With regards to readability, I see the following trade-off:
Using a template engine like Jinja allows to test the templating independently from the state execution. The output of the templates is YAML and tests could assert on expected output without necessarily be aware of the Salt context. (I personally like that, happy to develop further if useful.)
Do I personally find Jinja readable? It heavily depends on what it’s used for: Salt encourages keeping it as simple as possible (good rule of thumb for templates anyway), and I think that within that limit it’s an acceptable addition. Now, Jinja in a blank-space-meaningful markup like YAML doesn’t make for the most readable option either. (TL;DR: not really, no.)
Note: On second thought, the trade-off has to do with the choice of Jinja as templating engine for YAML, not Salt pillar itself.
I know it is possible to allow qubes to manage the state of other qubes via the Admin API, but haven’t looked much into it yet.
I disagree with almost everything that @restive says about Salt and its
place in Qubes.
There’s no expectation that minions will exist at execution time, and
there is a good system of dependencies - Salt has includes, state
ordering, a system of requisites including varieties of requires
and prereq. So it’s simple to ensure that a template/qube exists before running state.
Pillars can be used - imo one of the key reasons to use them is not
needed in Qubes. I don’t think the level of complexity introduced is that
hard to follow, but in the sorts of cases we are envisaging, I doubt they
are needed.
Most of the states at GitHub - unman/shaker started out as
examples for the notes at GitHub - unman/notes - they are
designed to be simple and easy to understand. The “Hello World” of
salting Qubes.
In any case, it doesn’t matter if the average user is baffled by Salt,
or cant understand pillars. They would probably baulk at the python used
in Qubes. At the moment they (mis)read documentation and blindly copy
shell scripts, or have someone set things up for them.
What we are talking about is a simple way for users to set up or add to
their Qubes system. We already use salt for this.
I think a simple solution is to package salt formulae in rpms and
distribute them using the standard mechanism that users use. It would be
good to provide a front end like apper, or one of the wire frames
produced years ago. If we did this right, then the command line in
whatever managementVM is used could drop out of use.
The questions then are:
How to vet and control what goes in to that repository - this would be a
question for any provisioning system.
How to keep reviewing and updating the packages and formulae (agnostic
as to sort)
How to produce a front end, or make the packages (whatever sort) easily
available.
I don’t want to shut down the question of whether salt is right for
Qubes, or some alternative might do better. But any solution will have
to deal with those last questions, and at the moment salt is what we
have.
We’ve been talking about this for 4-5 years - lets do something.
Happy to be wrong. It would save me time coming up with alternative solutions.
There is no expectation that the minions already exist in the Qubes version, how about the vanilla version, the idea I’m trying to get across is that Salt is not designed for the way that Qubes uses it and that creates problems that wouldn’t exist if Qubes was using something custom or something designed for a similar purpose.
Yes but how about dependencies on other formulas or cross vm dependencies?
I am not saying they are hard to follow like perl with goto statements is hard to follow, more like Java with too many abstract factory decorator proxy interface stub respositories. Overly verbose and time consuming to achieve having a vm name that can be changed in one place and that name reused throughout the formula.
Is that also true across formulas in Qubes?
Yeah, there is a fine line.
I must admit I do kind of love jinja. I just think the whole separate file tree shouldn’t be required for variables.
Yeah I have been thinking about working with that. I honestly wouldn’t mind spending some time putting something together to work with that API soon as I get some spare time (whenever that is).
My readability comments were more about things being longer than they need to be or would be in bash rather than about Salt being not usable by non technical users.
Do you envision those salt formulas continuing to run within Dom0? will the rpms automatically enable the tops? How would you handle naming conflicts and changes to the template vms?
I noticed the uniqueness requirement when duplicating states accross multiple formulas (e.g. to ensure a qube exists, say I want work to be a client for both split-SSH and split-GPG.)
That being said, and thinking about it now, it is not uncommon for file names to be state IDs (e.g. the first example in the salt.states.file docs). It doesn’t make sense that you couldn’t manage the same file in two machines, so I think I should amend the “globally”. It is very likely scoped by “execution environment” (no relation to the Salt envrionments, just putting words on the concept).
Two states with ID work-qube-is-present in dom0 is not OK, but two states with ID /etc/hostname in two different qubes is likely fine.
Disclaimer: I’m thinking through this, I haven’t checked, so take it with a grain of salt (no pun intended).
I am not sure I understand this part. I don’t believe Qubes OS modifies Salt to achieve that. (If someone actually knows, please chime in!)
It happens that one of the minions (dom0 is the Salt master, but behaves like a minion when state is asserted on it) has the ability to create other machines and their presence can be considered part of its state.
If asserting some state on those other machines is desirable (making them minions), ensuring their presence is merely a (regular) dependency between states, like the ones that require is meant to handle. I may be missing something, but don’t see why Salt would need to be customized for this.
Qubes OS does provides the qvm state module, but I don’t think that was the meaning behind “vanilla” vs “Qubes” Salt?
Am I in the ball park @restive, or missing the point of what you meant?
I currently use pillar to make the qubes names configurable (for example, a split-GPG formula defines the concept of gpg-client, but leaves to the user the ability to decide, via a configuration file, that they want their work and personal-email qubes to behave like gpg-clients).
This comment makes me wonder if there is a simpler way to achieve that. (This is probably not the thread to answer that specific question, but I think the example may provide some scope for the use of Salt pillar?)
Following the suggestion to re-center the conversation:
To the questions:
I would add:
How to provide guidance about how inter-dependent the packages (whatever they are) should be.
It seems to me that it makes sense that some packages build on each other, but also I think that such a repository would have less value if installing one package required to install a large number of them.
To go against my own strictures, I’ll comment on some of the salt specific
issues.
dependencies on other formulas - simply implemented in salt, using includes and varieties of requires
cross vm dependencies - Not sure how deep you want to dig. Ordinarily
you would mine, or use orchestrate, or a custom runner, or even have
minions interact directly with each other. In Qubes, none of this is
necessary because the managementVM has complete oversight and access to
all minions, grains and pillars.
If you mean “get configuration from qube A, use it in qube B”, then
since I would have salted the former it would be trivial to do the
latter.
If you want dependency on user generated configuration, the managementVM
can access that.
globally unique state IDs - unique at execution. It’s good practice to
avoid clash by using, e.g foo.configure and bar.configure instead of configure in foo and bar, and hoping for the best.
Bash/salt - I used to script Qubes - I don’t now. Whatever method is
used to set up systems will face the same issues in making formulae
available
General -
5. dom0? managementVM? Not sure of the relevance. Any scripts or setup
will require some management function.
6. Naming conflicts? Do you mean if someone creates their own “sys-net”
by hand? It should be straightforward to prompt user “I see you already
have foo - what name do you want to use for your shiny new qube?”. But these mechanisms could, if done right, stop all that.
7. Changes to templateVMs? I hope so. I am great advocate of multiple
templates, but whether we want to enforce this on users is a moot
point.
If some mechanism worked well, then it has the potential to transform
Qubes. Who cares if you have 80 qubes and 40 templates on your system?
People focus too much on the implementation, not the outcome, and the
Qube Manager and menu make that focus seem inevitable. It isn’t.
A naive user should be able to install a package that creates a
Template-printer and a Printer qube, with a menu item for “Print
settings”,and be able to use the printer as they want.
OK were you able to solve that problem? Ideally there should be a tag that should indicate whether the vm is a valid gpg client or something similar no? I wonder if you could add a tag to a vm, query that through the qubes pillar. How does jinja work with salt state naming I assume you can do a loop.index or some dictionary value from the thing you are looping as part of the name?
Also I guess it would probably work fine if there was some common package that needed to be installed by multiple formulas that could apply to the same VM.
Seems overkill to create environment per rpm but it may be the only way to make it all fit nicely together. I wonder if that would make it easier to sandbox the downloaded salt to ensure it has limited permissions.
Well .top files don’t exist in Salt, that’s purely a qubes thing, then also the way that VMs are started before state is applied, the order in which the application happens and the fact that you don’t need to start every VM to run the high state would be unique to Qubes I would imagine.
So in order for a formula to work the top needs to be enabled, this is not a thing in SaltStack, there is no top file so in order to require that another top is enabled you would need something custom to make that work.
My argument is that there should be but I think one would have to be created, which it wouldn’t if we used something that wasn’t salt. It’s like using a hammer to clean a window, yes you can extend the hammer to make it clean the window but the fact that you have to extend something to do something so fundamental to the activity you are engaged in is an indication that this is the wrong tool for the job.
OK, so how would my formula make use of your kali template? I would have to ask the user to first install your formula and enable the top before using it right? Unless my rpm depends on your rpm and then as part of the install of the RPM it ensures the top is enabled? Would that how you imagine it working?
Isn’t the whole point of qubes to limit the power that any one part of your system has to breach the security of the rest of it?
OK so in the same example I used above, what if I what I wrote depends on the Kali template you wrote, and there is a naming conflict, how do I depend on your Kali template when I can’t know for sure what the name of the template is going to be. Would it be tagged according to the RPM it came from so I know how to discover it?
I just mean that Qubes shouldn’t have a system for distributing these salt formulas that is worse than scripting them because it breaks if you have anything but the most vanilla install.
Focusing on the implementation is something you should do before implementing something right? I mean even if you were to decide to stop answering my questions, those questions would still have to be answered to implement the system you want to build right?
I have no idea if this is going to end in the right thread on the Forum,
or what it is going to look like, but here goes.
I’ve answered some of these questions before, I think.
OK were you able to solve that problem? Ideally there should be a tag that should indicate whether the vm is a valid gpg client or something similar no? I wonder if you could add a tag to a vm, query that through the qubes pillar. How does jinja work with salt state naming I assume you can do a loop.index or some dictionary value from the thing you are looping as part of the name?
Also I guess it would probably work fine if there was some common package that needed to be installed by multiple formulas that could apply to the same VM.
Uniqueness is an issue at execution, and easily resolved by sensible
naming : foo.config instead of config in foo.
It’s distinct from the targeting and requirement issue, which can be
solved, as you say by use of tags.
You can easily loop over elements if you have gone that way.
Seems overkill to create environment per rpm but it may be the only way to make it all fit nicely together. I wonder if that would make it easier to sandbox the downloaded salt to ensure it has limited permissions.
If you want to sandbox the salt, then run it in an managementVM where
you have set controls.
Well .top files don’t exist in Salt, that’s purely a qubes thing, then also the way that VMs are started before state is applied, the order in which the application happens and the fact that you don’t need to start every VM to run the high state would be unique to Qubes I would imagine.
Enabling a hierarchy of tops is a (painful and unnecessary) part of
salt. In many cases you can just state.apply states and they will do
the right thing™
So in order for a formula to work the top needs to be enabled, this is not a thing in SaltStack, there is no top file so in order to require that another top is enabled you would need something custom to make that work.
My argument is that there should be but I think one would have to be created, which it wouldn’t if we used something that wasn’t salt. It’s like using a hammer to clean a window, yes you can extend the hammer to make it clean the window but the fact that you have to extend something to do something so fundamental to the activity you are engaged in is an indication that this is the wrong tool for the job.
OK, so how would my formula make use of your kali template? I would have to ask the user to first install your formula and enable the top before using it right? Unless my rpm depends on your rpm and then as part of the install of the RPM it ensures the top is enabled? Would that how you imagine it working?
Installing states in an rpm is simple. The package creates the right
structure of a state tree under /srv, and in post-install executes the
state.
It’s at this point that you can interact with user to determine scope
or targets, and configure target qubes as necessary.
This does give the packages from whatever repo is in play significant
control, which is what we should be discussing in the other thread - it
affects any configuration mechanism, which is why it’s important to
get it right…
Isn’t the whole point of qubes to limit the power that any one part of your system has to breach the security of the rest of it?
OK so in the same example I used above, what if I what I wrote depends on the Kali template you wrote, and there is a naming conflict, how do I depend on your Kali template when I can’t know for sure what the name of the template is going to be. Would it be tagged according to the RPM it came from so I know how to discover it?
If you package depends on my kali package, then you do know what the
template is going to be. If it just depends on there being some
template called template-kali, (created by whatever means) then you
specify the requirement differently.
Maybe there is an argument for having certain reserved names (sys-
Template- etc)
I just mean that Qubes shouldn’t have a system for distributing these salt formulas that is worse than scripting them because it breaks if you have anything but the most vanilla install.
Focusing on the implementation is something you should do before implementing something right? I mean even if you were to decide to stop answering my questions, those questions would still have to be answered to implement the system you want to build right?
I meant the implementation of Qubes with templates, appVMs. The Manager
and default Menu makes people focus on those, but for many(most?) users
that focus is completely irrelevant.
You don’t need to know that Qubes uses salt to create a new template by
cloning an existing one, configuring the new template with packages,
creating a new appVM to hold the printer, creating a new Menu item where
you have to add the entry for “Printer settings”, and then configure
each qube to connect to that printer.
You don’t need to follow instructions that tell you to create a
template, connect it to the network, install software and configure
printer, and then change the templates for some qubes so you can print
from those qubes.
What you need to know is that you can install this package, and you
will have a printer settings item, and you can then print from your
applications. You do this - you can print.
Ironically, in many posts I rabbit on about increasing understanding and
forcing people to learn about the implementation - perhaps I’ve been
got at by Nina.
I hope I dont stop answering questions, time and other factors
permitting.
I can confirm that using RPM packages to install Salt states (and pillars if needed) doesn’t require complex packaging.
The nature of the RPM package is to mimic the target file system, and IMHO that makes placing files in /srv very readable.
My own packages install the states and pillar (when necessary) in /srv/user_salt and /srv/user_pillar. At this point, I’ve preferred not to execute post-install steps, but what @unman describes makes sense to me. (In my case you install the package, edit its config file at your pleasure, say /srv/user_pillar/split-ssh/config.yaml, and enable the top file(s). That is absolutely up to what kind of user interface you want, and I happen to be progressively automating further but only as I need it.
@restive I think part of what does make sense to you comes from the assumption / misunderstanding that Salt is modified in Qubes OS, e.g. top files.
I probably didn’t express myself clearly when I said it earlier (I cannot find my post in the reschufgle of the topic, so maybe its not there anymore, doen’t really matter) and @unman just said it too, but let me insist for clarity: top files are a Salt concept, that’s described in their docs.
Top files are what allow to apply (at once) a group of state files across multiple targets, where not all the state files apply to all targets. It a quite central concept to the use of Salt. That being said, you can do without if you want, they’re mostly a convenience when it comes to targeting. (I don’t personally find them painful, quite the opposite indeed, but YMMV.)
Does this make sense? (@unman linked to the docs above, and if you find my post it had the same link )
I didn’t suggest creating Salt environments oer-package or Salt formula. My choice of words wasn’t good.
This is a not-really-issue in my experience in practice, that can be solved by prefixing state IDs (as @unman suggests) or defining states only once and create dependencies (e.g. via Salt require) between formulas. Which you go for depends on your position in the DRY-and-modular - standalone spectrum. (Personally, I was leaning towards prefixing very much in the way @unman describes because so far I’d rather my formulas to be more standalone.) I don’t think any way is inherently wrong, rather it depends on who your target audience is and what you want the user experience to be.
I hope this helps clear some of your concerns @restive ?
Best not to rely on the good intentions of the developer of an RPM to respect the appropriate conventions, ideally prefix should be enforced.
I’m not sure that’s going to be easy, I imagine it’s going to be rather painful, but seems possible to do.
Any docs on how that works?
top.sls yes, *.top no
How do I guarantee your kali template is there without enabling the tops?
Sure that’s trivial, it’s just that those rpms are going to have the god mode in dom0, which is kind of weird for installing a kali template.
Yeah I think ideally it would run in a separate VM, with limited API access to the admin api or something but you mentioned that being in not a good state at present.
Yeah, I would much prefer an enforced prefix based on the name of the rpm or something to avoid conflicts
Yes except if you click install and it doesn’t allow you to print because of some modification you have made to your base template
I’m starting to run low on energy for this discussion sadly, not sure I’m effectively getting across my concerns.
top.sls exists in salt, .top doesn’t
I didn’t say you did, I was saying that it might be the only way to avoid naming conflicts which shouldn’t be a thing between code that you wrote and code that you didn’t write.
Do I have to edit other people’s code if they fail to use best practice and prefix their state ids? Relying on the good intentions and best practice to be followed seems pretty silly.
I’m not sure folks are fully understanding what I’m talking about as I don’t feel like the counter arguments address all of what I’m saying. It might be more efficient to have the discussion over voip, that might not be an option or of interest but thought I’d put it out there.
My understanding of the disagreement between @unman and me is that I think that while it’s possible to build a system like is needed using Salt there would be a lot of code smell and otherwise unnecessary complexity. @unman seems to argue that Salt is what exists in Qubes already, it is possible to implement the system with Salt therefor it should be implemented with Salt.
I don’t think @unman’s argument is without merit, despite my disagreement with the approach, it is possible to do this in Salt and it is possible to solve those problems that I raised with the existing system to create an app store.
I also appreciate that I’m new here and coming in here telling you guys what you are doing wrong isn’t a great look. I do want to say that I really admire the work you’ve done and for an amazing cause in case I didn’t put that across.
I think it might be an idea to take a new approach to the discussion what are the requirements for this system, and what work would be involved in adapting what exists already to meet these requirements?
Here would be my list in no particular order of the top of my head.
User should be able to read the code before execution
User should be able to approve what the permissions the “app” will use before execution
Apps should be able to depend resources created by other Apps reliably
User shouldn’t have to deal with naming conflicts
App developer shouldn’t have to deal with naming conflicts
Apps should be able to add or update a policy in Qubes without having access to add or update every policy in Qubes
Apps should have the ability to update templates without having the ability to update every template
User should have the ability to choose the templates used by Apps if they have multiple supported templates installed
User shouldn’t have to alter the state of their system to install an app
User should be able to create their own App repository
User should be able to uninstall the App and have all resources created by that app removed
User should be able to update Apps
Apps should be able to define relationships between domains created or request relationships to other domains
Users should be able to install apps in a low trust mode to try before you buy
Users should be able to install the same App multiple times
Best not to rely on the good intentions of the developer of an RPM to respect the appropriate conventions, ideally prefix should be enforced.
No one will rely on anything except the review by the person
curating whatever distribution method is agreed.
I’m not sure that’s going to be easy, I imagine it’s going to be rather painful, but seems possible to do.
When I said “easy” to loop over elements I meant trivial.
Any docs on how that works?
The best intro to managementVM is still at
top.sls yes, *.top no
I gave a reference to the use of top files in salt - I dont understand
why you are still arguing this. If you use salt at all you will know that
top files are widely used.
How do I guarantee your kali template is there without enabling the tops?
Sure that’s trivial, it’s just that those rpms are going to have the god mode in dom0, which is kind of weird for installing a kali template.
God mode? Do you think the same when installing packages from fedora or
Qubes? I feel we are talking past each other. There is no greater risk
here.
Yeah I think ideally it would run in a separate VM, with limited API access to the admin api or something but you mentioned that being in not a good state at present.
I dont recall saying this, and am to tired to listen back. It’s perfectly
workable and I use a managementVM regularly.
What you need to know is that you can install this package, and you
will have a printer settings item, and you can then print from your
applications. You do this - you can print.
[/quote]
Yes except if you click install and it doesn’t allow you to print because of some modification you have made to your base template
If you’re capable of modifying your templates, then you probably dont
need a packaged solution.
But a packaged salt formula can enforce the requirements in (some)
template - that’s the point.
I’m starting to run low on energy for this discussion sadly, not sure I’m effectively getting across my concerns.
I agree that top files are a thing in Salt, there is only one top file, it is called top.sls. Qubes adds files that can be named blah.top. Files with the extension .top do not exist in vanilla Salt.
but also the part about Salt being supported is pretty rad.
)