Design and create salt formulas distribution method (#1939)

Hi Gonzalo! The core dev team is working hard right now at squashing bugs and getting 4.1 out the door, I believe. I believe.

There was a thread already on GitHub about release frequency/cadence and priorities, that you may have commented on already. In that thread, I raised a similar issue as what you identified here—what are the priorities for the project overall, and how can we give better visibility to contributors on those things… what’s ahead, etc. Once 4.1 has been released and everyone is breathing a little easier, my user testing & design for the AppMenu redesign project have also been wrapped, I’d like to do a workshop with the whole team (if Marek feels it’d be helpful, and as others have availability) to establish some processes to keep this kind of stuff going and public somewhere.

TL;DR, it may be a few months, but I think we all want the same things—it’s on our radar, and on our radar to do well! Just scrambling to make sure a few items that are almost fully cooked and nearing the burn point, don’t actually burn and make it to the table, first. :slight_smile:

@Sven Yes, everything makes sense! The clarification on current state for the Salt formula distribution issue, the description of the problem to be solved, and on a process level, the role of this thread and the GitHub issue :+1:

@ninavizz I remember you raised similar process/coordiantion questions some time ago, indeed. I’m happy to hear that you’ve been heard and those are on everyone’s radar. :slightly_smiling_face:

You folks got this! (And that’s all I needed to hear!) Thanks both for taking the time to clarify and explain, I really appreciate it.

With regards to timing, yep, things may take time, that’s what prioritization is about. I’ll be around if contributor-user feedback is needed, and in the meantime I’ll stay around (in this thread) in case I can help with the specific question of reviewing and distributing Salt formulas.

Yes, this is a good summary of the issue.

Speaking as a Qubes user, I don’t realistically see any way to make a Salt formula sufficiently trustworthy other than (1) being able to understand it myself or (2) a verifiable signature from the core Qubes team.

The problem with (1) is that Salt is extremely difficult to learn (at least it seems so to me; I once tried and didn’t make much progress). The problem with (2) is that it demands precious time from the core devs, especially Marek. Having Marek review every submitted Salt formula simply isn’t scalable.

If some weaker method is used, e.g., letting people rate Salt formulas on a five-star scale, then I’ll simply ignore this new Salt store and just keep using my own simple Bash scripts. They’ve served me well enough throughout my years of using Qubes, and I don’t need anything fancy. If anything, I’m trying to simplify my computing dependencies, not make them more complex. Likewise, if the Salt formulas were reviewed by non-Qubes-core security experts, I’d take a pass, since the risks associated with increasing my circle of trust outstrip the potential benefits I’d gain. To paraphrase Buffett, I’d be risking something I have and need for something I don’t have and don’t need, and that’d be foolish.

2 Likes

Someone suggested creating a system that analyzes the Salt formula and determines what it can do or which permissions it requires, similar to the way browser extensions and mobile apps often work. That could go a long way toward automating the “trust problem” (since people could choose to gamble with low-risk Salt formulas), but it wouldn’t remove the need for review. If anything, the history of app/extension store malware shows that users expect everything in the store to be safe, and they’ll blame us if it isn’t. The bar of expectation is no doubt even higher for a security-oriented OS like Qubes.

That’s an interesting idea!

Unfortunately I have to agree with all of the above.

So how do we make Qubes OS easier to use for people that are not comfortable installing software using the command line or spending man months on learning how to configure the intricacies of Linux?

Maybe a “salt store” is the wrong perspective. Maybe we need to limit what we want to do to a certain set of use cases like:

  • create specific application qubes (e.g. signal qube, libreoffice qube, thunderbird qube, etc.)
  • create specific system qubes (e.g. vpn qube, pi-hole qube, davmail qube, etc.)
  • apply the theming, font, icon and DPI choices made in dom0 to all linux based qubes
  • create disposable qubes for web browsing, offline media consumption, offline editing
  • etc.

… and yes these all would need to be ultimately verified, owned and signed by the core team. What other way is there?

More clearly and to the point: I see a certain set of features that are desperately needed to make Qubes OS more accessible to “normal” users. These features (looking at them) will most likely be implemented as salt formula. I don’t care if they are in a “store”.

Maybe this is the same thing more or less as submitting any other kind of feature to Qubes OS. You make a pull request and at some point Marek will take a look and decide if that’s something that should exist and if it’s implemented correctly.

Finally: since this is not trivial I think the amount of PR’s will probably be manageable.

1 Like

Well to add a bit to this discussion, I envision the salt distribution more of a thing for organizations. For example a newsroom has its tech staff develop an formula which may or not be shared externally. I don’t see it scaling safely in any other way.

So with that in mind, my suggestion for thinking about any salt distribution mechanism would be to make it easier to distribute formulas across an organization. Something that provides update notifications, for example or lets the user visually run a salt formula configured by their systems administrator.

1 Like

Same for me. And I must say I also found getting started learning Salt (and making sense of it in the context of Qubes OS) required a significant effort. (Enjoyable effort, but significant nontheless!)

A bit of context: writing down configuration to automate complex interactions is something that helps me a lot to making sense of them. That prompted me to start publishing my learning notes (forum thread) and reflect on how I would explain how to use Salt for Qubes OS to someone already comfortable with the tasks they want to automate. (That’s a key part where I agree totally with the thoughts that @adw and @Sven posted above.)

To build on the above, my takeaways so far:

  1. There are a few recurrent patterns that are specific to Qubes OS.
  2. Even if it’s not obvious at first, using Salt in Qubes OS isn’t that different from using Salt otherwise because the Qubes OS team provides tools to manage the qubes themselves.
  3. Salt is extremely flexible, its docs reflect that, and that makes it harder to get started. I am convinced that providing opinionated ways to render a few recurrent patterns can provide guidance to decompose complex scenarios in a collection of simpler parts. Once that done, in my experience, the Salt concepts handle the coordination of smaller pieces nicely, the official Salt docs become very useful reference material and the resulting states/formulas can be quite readable.
  4. Automated testing of Salt states and formulas requires a significant setup. I haven’t settled on anything satisfactory yet myself.

I’ve been able to create a range of states and formulas from the following limited set of patterns.

I’ll use “qube” generically for brevity, some of the patterns make sense for an AppVM or a TemplateVM or dom0 or several of them but the idea is the same:

  • ensuring that a qube exists
  • ensuring that a package is available in a qube
  • ensuring that a file is present in a persitent location of a qube
  • ensuring that a file is present in a non-persistent location of a qube (that is, gets created when the qube starts)
  • ensuring that some content is present in a persistent file in a qube
  • ensuring that some content is present in a non-persistent file in a qube (that is, gets added to the file when the qube starts)

Each of those patterns, in turn, can be rendered quite simply in Salt, which is good for review. (Happy to provide examples for discussion if useful.)

That’s what I have so far. To recap: I don’t think how we can skip on formulas being reviewed by the Qubes OS team, or people being able to understand the formulas themselves. That being said, I believe we can decompose most formulas into a collection of well-known patterns (see above for a starting point), which could be a step towards making review less time-consuming and the Salt + Qubes OS learning curve more accessible.

1 Like

Speaking as a Qubes user, I don’t realistically see any way to make a Salt formula sufficiently trustworthy other than (1) being able to understand it myself or (2) a verifiable signature from the core Qubes team.

The problem with (1) is that Salt is extremely difficult to learn (at least it seems so to me; I once tried and didn’t make much progress). The problem with (2) is that it demands precious time from the core devs, especially Marek. Having Marek review every submitted Salt formula simply isn’t scalable.

If some weaker method is used, e.g., letting people rate Salt formulas on a five-star scale, then I’ll simply ignore this new Salt store and just keep using my own simple Bash scripts. They’ve served me well enough throughout my years of using Qubes, and I don’t need anything fancy. If anything, I’m trying to simplify my computing dependencies, not make them more complex. Likewise, if the Salt formulas were reviewed by non-Qubes-core security experts, I’d take a pass, since the risks associated with increasing my circle of trust outstrip the potential benefits I’d gain. To paraphrase Buffett, I’d be risking something I have and need for something I don’t have and don’t need, and that’d be foolish.

I’ve been a little preoccupied elsewhere.

Andrew said:

  1. Salt is extremely difficult to learn
  2. Any store or distribution mechanism would require verifiable signature from core team.

On the first, I think it isn’t, at least in Qubes. The basic mechanisms
are no harder to understand than use of qvm- tools. I have notes at
GitHub - unman/notes which aim to make use in Qubes simple,
and Gonzalo has written an introduction.

In any case, the point is that end users wouldn’t have to understand
salt
to get particular configurations on their machines, any more than
they need to understand salt to provision the machine on start up or to
run updates.

To make an obvious point, users seem quite content to run arbitrary
bash scripts when they clearly don’t understand what the effects are.
This should be better than that.

Agreed Marek shouldn’t be adding to his current 48 hr/day workload. But
because Salt states and formulae are straightforward, they don’t take a
huge amount of scrutiny. I’m tempted to say that if you cant
understand a salt state at first sight it’s not well written.

I’ll be putting a repository online in the next few days with assorted
packages aimed at generating outcomes as poc.
It’s already true that there are packages in the main repositories that
users aren’t aware of, and rather than creating a store, adapting apper
in dom0 to make it easy to find those packages (salt or not) would be a
further step away from the command line.

2 Likes

Probably a combination of good defaults and easy GUI wrappers.

This is a good idea and could certainly streamline (or partially automate) the review process.

That’s good news. Those resources didn’t exist when I tried to learn it.

Of course. I don’t think this was ever in question. However, we also do not want to condone the activity of running untrusted Salt formulas, even if people run arbitrary bash scripts. We want to discourage both.

I think this depends on the referent of “you.” This certainly applies to folks like you and Marek, but I doubt it would apply to non-technical users, probably not even the average Qubes user. Nonetheless, a reason for hope.

1 Like

Hey, I’m a new user, and I have to say I’m blown away by Qubes, it’s a lot more user friendly than I expected it to be given the level of security that is possible with this system.

Regarding being inclusive of non technical users I would doubt non technical people install or setup Qubes by themselves, for a Journalist they likely have their IT folks involved in setting up the system. I wonder if part of the barrier to entry for some non technical users is actually budgeting engineer time to install and configure the OS. I don’t know if they typically hire outside contractors who specialise in Qubes installations or if they usually ask their general IT folks to install it, but if it is the general IT folks doing a small number of installs then even though it’s a technical user the UI and time commitment is still rather important.

Even as a technical user getting going with Qubes has been a serious time commitment, it’s not that I couldn’t get the operating system working just like any other linux distro in a similar amount of time but rather that if I want to benefit from the features of Qubes there is a lot of work involved in setting all of that up. Then if for some reason my yubikey breaks or my laptop dies I’m going to have to do all of that setup again, even if I have backups created that only covers the home directory.

Ideally there would be a list of developer maintained and (for the brave) a community supported repository of one click configurations like split-gpg with thunderbird or similar. This could actually increase security for users by reducing the barrier to entry for some of the more complex security configurations.

I dislike SaltStack in Qubes, learning SaltStack is quite a time commitment by itself and then you have to understand the differences that Qubes brings to the table. I’m not sure that Qubes leverages enough of the advantages of SaltStack for it to add real value. I think that becomes pretty obvious when you try to imagine what parameterising SaltStack would look like to avoid collisions with existing configuration. Square peg, round hole.

I wonder if a better arrangement might be something custom, something similar to docker compose that just defines the vm definition, relationships to other vms in the compose file and the bash script to run inside each vm on init?

3 Likes

12 posts were merged into an existing topic: SaltStack Challenges and Experiences

We are wildly off topic and I suspect that many people who might have
contributed now wont.

Pulling it back to the main topic - It wouldn’t be unreasonable to prompt
users on package install for certain information. E.g a VPN package
prompts as to whether all or specific qubes should use the VPN.

On interdependencies, look at the GitHub discussions of Qube flavors. No
reason why there shouldn’t be metapackages for complete out of the box
customisation.
Most repositories or package managers already have collections of some
sort. The description should explain what the Qubes system will look
like after install.

This discussion could be implementation agnostic - it could be bundles of
bash scripts.

Just moved 17 posts onto a discussion which I called SaltStack Challenges and Experiences. Feel free to continue here on topic or the other thread for more staltstack experiences.

1 Like

Quoting myself for context:

(This is probably not the thread to answer that specific question, but I think the example may provide some scope for the use of Salt pillar.)

Following the suggestion to re-center the conversation:

I would add [to @unman’s questions]:

  • How to provide guidance about how inter-dependent the packages (whatever they are) should be.

It seems to me that it makes sense that some packages build on each other, but also I think that such a repository would have less value if installing one package required to install a large number of them. (original)

:confused: I certainly didn’t think of demotivating people to join the conversation by addressing some Salt-specific questions I found legitimate from the previous posts, I hope my intention at least was clear. Apologies if I did, I’ll back off.

There’s a major hurdle in distributing salt packages using an rpm -
everyone says that you cant get interaction using an rpm.
This isn’t strictly true - you can hack the %post section to do some
interaction, and this works at the command line.
But since part of the aim here is to move away from command line use,
that’s not the solution.

I’ve been playing around with installing a simple script, and triggering
it in %post - the script would prompt (e.g):

Which qubes do you want to use the multimedia-player?
All - Some*select from drop down?* - I'll configure myself

and then salt using the newly installed formula accordingly - anyone have
thoughts on this?

On another front I’ve been hacking apper to only provide information
from specific repositories (to avoid offering options to install other
stuff in dom0), and to work properly with qubes-dom0-update.
That could be neat solution rather than building a Qubes specific
solution from scratch.

Here’s my current thinking about providing some sort of app-store.
I’m going to reference salt, but the same would apply to other
mechanisms.

I like the idea of providing states using rpm packages:
they are easily signed, for authentication purposes; there would be
minimal set-up required; they fit with existing methods.

Interaction with existing?

  1. Don’t override current configuration, unless that is purpose of
    package.
  2. Work with current defaults,(e.g default template), unless good reason not
    to.
  3. Don’t download a template unless that is explicitly part of package
    install, or explained in description.
  4. Don’t assume - a user may delete debian-10 template and create a Fedora
    qube called debian-10.
  5. Don’t surprise.
  6. If install requires user intervention - e.g. selecting qubes to which
    a change should be applied - provide a basic tool for this. - How should
    this be signalled to user?
  7. If an install requires user configuration - e.g. placing an openvpn
    config file - how should user be told what to do, and how to do it?

If you are using a configuration tool like salt,(or ansible, or bash
scripts, or …), install if you can.
If you cant install, leave the formula in a known place.
An informed user may want to edit the module and then install state.

Packages for the app store must be reviewed before inclusion.
Packages must be easily understandable.

Should removing the package remove the build? I don’t know. That sounds
sensible, but could have side effects depending on what the user has
done post install.

Thoughts?

Just share your samples w/ github. Think there’re more people like me that prefer to learn by sample. If you put a readme (i.e. quick howto how to apply the sample) - even better.
At least I’ve got a reason to get into salt - too many work on Qubes OS reinstall - some should be done by automation.

With rpm you have to definitely trust the rpm builder since anything could be placed in install script. And it’s not very handy to check what rpm does - require some digging. If the rpm w/ salt stuff is signed by Qubes OS team that’s definitely a reason to trust. I’ve seen you provide samples w/ salt stuff. Is there something that is already signed by Qubes OS team?

Agree.

You should copy the default template & modify copy. It’s not a good idea to alter default template - if something gets wrong you cannot replace it from backup-clone - you have to reinstall default template w/ rpm. Also default template may be already used for many app qubes & I think it’s better to ask user to alter template value for his/her app vm not to silently apply a new template w/o user intervention.

Agree

Agree

Agree.

Depends to the way the software installed - terminal or gui. It should take a list of qubes present & allow user to select it either by menu either by entering a number pointing to existing qube. In gui it should be a menu. In terminal - numbers will be enough. This is not possible w/ RPM w/o hacks, but maybe it is possible to call external utility from rpm? At least I guess rpm installation should provide ability to get things in some order. So external tool should just not return till user response.

external to rpm tool needed. One possible solution - It could write user response to files & these files then will be parsed by next command inside rpm.

That would be even better - two-step installs. 1st step places the formula for salt & whatever script, user reviews it and then executes apply step. Sounds good. :slight_smile: Then both install and apply utilities could be verified and distributed by Qubes team. People just put recipes in easy readable form & users will have to trust less for third party recipes (easier to check than ready to use rpms)

This should be controlled by an option controlled by user.

I would be glad to have a video tutorial. :slight_smile: At least instead of reading tons of documentation I’d like to have many samples in github & one-two videos to get started as quick as possible. Well, the problem w/ videos is that is could be done only w/ external camera - no one would like to install in their Dom0 full software for screen capture with huge dependencies.

@grey_olli My Qubes Os-related Salt formulas and states are available on GitHub already, along with the tooling I use to package them as RPM. If you follow the forum link in the post you quoted, you’ll quickly find references to my notes and the corresponding GitHub repositories. :slightly_smiling_face: