Design and create salt formulas distribution method (#1939)

[Sven]

Nina wrote :

I see a LOT of time spent by folks, rabbit-holing on Issues here in
GitHub, that do not seem to align with actual priorities for the
QubesOS project to succeed beyond the walls of its existing
community.

Design and create salt formulas distribution method

I see this too and definitely contribute to it. Maybe we can help in
such a case by starting a parallel thread here and posting a link to it
to the Github issue.

This would reserve the Github discussion for technical communication
between reporter and developer, while everyone else who has an idea or
wants to contribute can do so in the forum. Since it’s linked both ways
contributors can still read the comments (in the forum) if they wish,
while we keep the Github communication more focused on the ‘doing stuff’
part.

@adw would you be OK with that? It would allow you to enforce focus on
GitHub also a bit more aggressively by redirecting folks to the forum.

[gonzalo-bulnes]

Hi @Sven! I’ll ask the question naively in the hopes that it’s useful. If it’s not, please disregard.

If there is already a solution being worked on (on GitHub), isn’t it somewhat late to determine if solving the problem now aligns with the current priorities?

I love the idea of opening a thread to discuss problems, define them, understand their impact then discuss priorities and possible solutions.

I do believe that the forum is probably a better place than GutHub to do that because I think it is more accessible to a more diverse crowd. (The forum is also where some (a lot of?) problem symptoms are first raised, which may prove convenient.)

I have no doubt that when the time comes to work on a solution, the GitHub issue will be more useful if it can be focused the technical aspects (the how) because the problem to solve has been scoped (the what and who, considering there is the Qubes OS team and contributors) and its priority established (the when) in the forum.

That being said, all that assumes that the conversations happens before the work on a solution is started.

Additional (but related) question: when it comes to finding out what priority to assign to any given problem, is there any place where the current goals are published?

I suppose there are roadmaps somewhere, but don’t know if they are not public already. In my opinion, having a public list of priorities to refer to would be helpful when finding out if a given piece of work is more suited for community contributions or should be driven mainly by the Qubes OS team. And maybe once decisions are made contributors can refer to that same list in order to ensure their constibutions are effective. (That’s the part that interests me the most at the moment )

[adw]

@Sven

This would reserve the Github discussion for technical communication
between reporter and developer, while everyone else who has an idea or
wants to contribute can do so in the forum. Since it’s linked both ways
contributors can still read the comments (in the forum) if they wish,
while we keep the Github communication more focused on the ‘doing stuff’
part.

@adw would you be OK with that? It would allow you to enforce focus on
GitHub also a bit more aggressively by redirecting folks to the forum.

Yes, this is actually our policy. I just don’t enforce it as strictly as I’d like, because that would drive everyone away from the project.

I’ll clarify this further in the documentation.

[Sven]

If there is already a solution being worked on (on GitHub), isn’t it
somewhat late to determine if solving the problem_now_ aligns with
the current priorities?

There is nothing actually being worked on (yet). Nina would take care of
the UI/UX of a potential solution and pointed out that there are plenty
of things to do that would have more immediate impact, hence higher
priority.

At the same time “design and create salt formulas distribution method”
is mostly a process / tools question with the UI “just” being the part
presenting the end result (and maybe in some far future support the
process).

However, we first have to find the process / solution for: how can
community members contribute salt formula? And: how can these be
reviewed so users who do not understand salt formula can make trust
decisions and use them?

We need to discuss, start using and iterate probably on how to do this.

@unman is already providing a repository he calls “shaker” and him being
a long time member and core team member the trust question in his case
isn’t that urgent. But imagine e.g. I would post a salt recipe… how
would you know that you can use it without harm? Maybe I made a mistake
that can brick your system? Maybe I am a malicious actor? … so you
would want many other people review the formula and give feedback.

How to do this and how to present it and also how and in which form to
deliver those formula … that’s what needs to be discussed.

This being an important topic many community members will have ideas and
opinions and the forum is much better to have this discussion. Here is
where we can “go down rabbit holes”, branch into different threads and
can discuss as much as we want.

GitHub threads are meant to be tactical exchange between contributors
and reviewers/testers. It’s not the place to have a long discussion.

So I am hoping we can find a way to be more respectful of the
developer’s attention and move much of the conversation here and just
link the thread to/from GitHub. That way the developers will still see
it but their Inbox won’t get flooded with notifications about things
that are less relevant to their current work.

Makes sense?

[ninavizz]

Hi Gonzalo! The core dev team is working hard right now at squashing bugs and getting 4.1 out the door, I believe. I believe.

There was a thread already on GitHub about release frequency/cadence and priorities, that you may have commented on already. In that thread, I raised a similar issue as what you identified here—what are the priorities for the project overall, and how can we give better visibility to contributors on those things… what’s ahead, etc. Once 4.1 has been released and everyone is breathing a little easier, my user testing & design for the AppMenu redesign project have also been wrapped, I’d like to do a workshop with the whole team (if Marek feels it’d be helpful, and as others have availability) to establish some processes to keep this kind of stuff going and public somewhere.

TL;DR, it may be a few months, but I think we all want the same things—it’s on our radar, and on our radar to do well! Just scrambling to make sure a few items that are almost fully cooked and nearing the burn point, don’t actually burn and make it to the table, first.

[gonzalo-bulnes]

@Sven Yes, everything makes sense! The clarification on current state for the Salt formula distribution issue, the description of the problem to be solved, and on a process level, the role of this thread and the GitHub issue

@ninavizz I remember you raised similar process/coordiantion questions some time ago, indeed. I’m happy to hear that you’ve been heard and those are on everyone’s radar.

You folks got this! (And that’s all I needed to hear!) Thanks both for taking the time to clarify and explain, I really appreciate it.

With regards to timing, yep, things may take time, that’s what prioritization is about. I’ll be around if contributor-user feedback is needed, and in the meantime I’ll stay around (in this thread) in case I can help with the specific question of reviewing and distributing Salt formulas.

[adw]

@Sven

However, we first have to find the process / solution for: how can
community members contribute salt formula? And: how can these be
reviewed so users who do not understand salt formula can make trust
decisions and use them?

We need to discuss, start using and iterate probably on how to do this.

@unman is already providing a repository he calls “shaker” and him being
a long time member and core team member the trust question in his case
isn’t that urgent. But imagine e.g. I would post a salt recipe… how
would you know that you can use it without harm? Maybe I made a mistake
that can brick your system? Maybe I am a malicious actor? … so you
would want many other people review the formula and give feedback.

How to do this and how to present it and also how and in which form to
deliver those formula … that’s what needs to be discussed.

Yes, this is a good summary of the issue.

Speaking as a Qubes user, I don’t realistically see any way to make a Salt formula sufficiently trustworthy other than (1) being able to understand it myself or (2) a verifiable signature from the core Qubes team.

The problem with (1) is that Salt is extremely difficult to learn (at least it seems so to me; I once tried and didn’t make much progress). The problem with (2) is that it demands precious time from the core devs, especially Marek. Having Marek review every submitted Salt formula simply isn’t scalable.

If some weaker method is used, e.g., letting people rate Salt formulas on a five-star scale, then I’ll simply ignore this new Salt store and just keep using my own simple Bash scripts. They’ve served me well enough throughout my years of using Qubes, and I don’t need anything fancy. If anything, I’m trying to simplify my computing dependencies, not make them more complex. Likewise, if the Salt formulas were reviewed by non-Qubes-core security experts, I’d take a pass, since the risks associated with increasing my circle of trust outstrip the potential benefits I’d gain. To paraphrase Buffett, I’d be risking something I have and need for something I don’t have and don’t need, and that’d be foolish.

[adw]

Someone suggested creating a system that analyzes the Salt formula and determines what it can do or which permissions it requires, similar to the way browser extensions and mobile apps often work. That could go a long way toward automating the “trust problem” (since people could choose to gamble with low-risk Salt formulas), but it wouldn’t remove the need for review. If anything, the history of app/extension store malware shows that users expect everything in the store to be safe, and they’ll blame us if it isn’t. The bar of expectation is no doubt even higher for a security-oriented OS like Qubes.

[deeplow]

@adw

Someone suggested creating a system that analyzes the Salt formula and determines what it can do or which permissions it requires, similar to the way browser extensions and mobile apps often work.

That’s an interesting idea!

[Sven]

@adw

[…] I don’t realistically see any way to make a Salt formula sufficiently trustworthy other than […] verifiable signature from the core Qubes team

[…] having Marek review every submitted Salt formula simply isn’t scalable

[…] if the Salt formulas were reviewed by non-Qubes-core security experts, I’d take a pass, since the risks associated with increasing my circle of trust outstrip the potential benefits

[…] the history of app/extension store malware shows that users expect everything in the store to be safe, and they’ll blame us if it isn’t.

Unfortunately I have to agree with all of the above.

So how do we make Qubes OS easier to use for people that are not comfortable installing software using the command line or spending man months on learning how to configure the intricacies of Linux?

Maybe a “salt store” is the wrong perspective. Maybe we need to limit what we want to do to a certain set of use cases like:

• create specific application qubes (e.g. signal qube, libreoffice qube, thunderbird qube, etc.)
• create specific system qubes (e.g. vpn qube, pi-hole qube, davmail qube, etc.)
• apply the theming, font, icon and DPI choices made in dom0 to all linux based qubes
• create disposable qubes for web browsing, offline media consumption, offline editing
• etc.

… and yes these all would need to be ultimately verified, owned and signed by the core team. What other way is there?

More clearly and to the point: I see a certain set of features that are desperately needed to make Qubes OS more accessible to “normal” users. These features (looking at them) will most likely be implemented as salt formula. I don’t care if they are in a “store”.

Maybe this is the same thing more or less as submitting any other kind of feature to Qubes OS. You make a pull request and at some point Marek will take a look and decide if that’s something that should exist and if it’s implemented correctly.

Finally: since this is not trivial I think the amount of PR’s will probably be manageable.

[deeplow]

Well to add a bit to this discussion, I envision the salt distribution more of a thing for organizations. For example a newsroom has its tech staff develop an formula which may or not be shared externally. I don’t see it scaling safely in any other way.

So with that in mind, my suggestion for thinking about any salt distribution mechanism would be to make it easier to distribute formulas across an organization. Something that provides update notifications, for example or lets the user visually run a salt formula configured by their systems administrator.

[gonzalo-bulnes]

@adw

Speaking as a Qubes user, I don’t realistically see any way to make a Salt formula sufficiently trustworthy other than (1) being able to understand it myself or (2) a verifiable signature from the core Qubes team.

Same for me. And I must say I also found getting started learning Salt (and making sense of it in the context of Qubes OS) required a significant effort. (Enjoyable effort, but significant nontheless!)

A bit of context: writing down configuration to automate complex interactions is something that helps me a lot to making sense of them. That prompted me to start publishing my learning notes (forum thread) and reflect on how I would explain how to use Salt for Qubes OS to someone already comfortable with the tasks they want to automate. (That’s a key part where I agree totally with the thoughts that @adw and @Sven posted above.)

To build on the above, my takeaways so far:

1. There are a few recurrent patterns that are specific to Qubes OS.
2. Even if it’s not obvious at first, using Salt in Qubes OS isn’t that different from using Salt otherwise because the Qubes OS team provides tools to manage the qubes themselves.
3. Salt is extremely flexible, its docs reflect that, and that makes it harder to get started. I am convinced that providing opinionated ways to render a few recurrent patterns can provide guidance to decompose complex scenarios in a collection of simpler parts. Once that done, in my experience, the Salt concepts handle the coordination of smaller pieces nicely, the official Salt docs become very useful reference material and the resulting states/formulas can be quite readable.
4. Automated testing of Salt states and formulas requires a significant setup. I haven’t settled on anything satisfactory yet myself.

I’ve been able to create a range of states and formulas from the following limited set of patterns.

I’ll use “qube” generically for brevity, some of the patterns make sense for an AppVM or a TemplateVM or dom0 or several of them but the idea is the same:

• ensuring that a qube exists
• ensuring that a package is available in a qube
• ensuring that a file is present in a persitent location of a qube
• ensuring that a file is present in a non-persistent location of a qube (that is, gets created when the qube starts)
• ensuring that some content is present in a persistent file in a qube
• ensuring that some content is present in a non-persistent file in a qube (that is, gets added to the file when the qube starts)

Each of those patterns, in turn, can be rendered quite simply in Salt, which is good for review. (Happy to provide examples for discussion if useful.)

That’s what I have so far. To recap: I don’t think how we can skip on formulas being reviewed by the Qubes OS team, or people being able to understand the formulas themselves. That being said, I believe we can decompose most formulas into a collection of well-known patterns (see above for a starting point), which could be a step towards making review less time-consuming and the Salt + Qubes OS learning curve more accessible.

[unman]

Speaking as a Qubes user, I don’t realistically see any way to make a Salt formula sufficiently trustworthy other than (1) being able to understand it myself or (2) a verifiable signature from the core Qubes team.

The problem with (1) is that Salt is extremely difficult to learn (at least it seems so to me; I once tried and didn’t make much progress). The problem with (2) is that it demands precious time from the core devs, especially Marek. Having Marek review every submitted Salt formula simply isn’t scalable.

If some weaker method is used, e.g., letting people rate Salt formulas on a five-star scale, then I’ll simply ignore this new Salt store and just keep using my own simple Bash scripts. They’ve served me well enough throughout my years of using Qubes, and I don’t need anything fancy. If anything, I’m trying to simplify my computing dependencies, not make them more complex. Likewise, if the Salt formulas were reviewed by non-Qubes-core security experts, I’d take a pass, since the risks associated with increasing my circle of trust outstrip the potential benefits I’d gain. To paraphrase Buffett, I’d be risking something I have and need for something I don’t have and don’t need, and that’d be foolish.

I’ve been a little preoccupied elsewhere.

Andrew said:

1. Salt is extremely difficult to learn
2. Any store or distribution mechanism would require verifiable signature from core team.

On the first, I think it isn’t, at least in Qubes. The basic mechanisms
are no harder to understand than use of qvm- tools. I have notes at
GitHub - unman/notes which aim to make use in Qubes simple,
and Gonzalo has written an introduction.

In any case, the point is that end users wouldn’t have to understand
salt to get particular configurations on their machines, any more than
they need to understand salt to provision the machine on start up or to
run updates.

To make an obvious point, users seem quite content to run arbitrary
bash scripts when they clearly don’t understand what the effects are.
This should be better than that.

Agreed Marek shouldn’t be adding to his current 48 hr/day workload. But
because Salt states and formulae are straightforward, they don’t take a
huge amount of scrutiny. I’m tempted to say that if you cant
understand a salt state at first sight it’s not well written.

I’ll be putting a repository online in the next few days with assorted
packages aimed at generating outcomes as poc.
It’s already true that there are packages in the main repositories that
users aren’t aware of, and rather than creating a store, adapting apper
in dom0 to make it easy to find those packages (salt or not) would be a
further step away from the command line.

[adw]

@Sven

So how do we make Qubes OS easier to use for people that are not comfortable installing software using the command line or spending man months on learning how to configure the intricacies of Linux?

Probably a combination of good defaults and easy GUI wrappers.

@Sven

Maybe we need to limit what we want to do to a certain set of use cases […]

This is a good idea and could certainly streamline (or partially automate) the review process.

@unman

On the first, I think it isn’t, at least in Qubes. The basic mechanisms
are no harder to understand than use of qvm- tools. I have notes at
GitHub - unman/notes which aim to make use in Qubes simple,
and Gonzalo has written an introduction.

That’s good news. Those resources didn’t exist when I tried to learn it.

@unman

In any case, the point is that end users wouldn’t have to understand
salt to get particular configurations on their machines, any more than
they need to understand salt to provision the machine on start up or to
run updates.

To make an obvious point, users seem quite content to run arbitrary
bash scripts when they clearly don’t understand what the effects are.
This should be better than that.

Of course. I don’t think this was ever in question. However, we also do not want to condone the activity of running untrusted Salt formulas, even if people run arbitrary bash scripts. We want to discourage both.

@unman

But because Salt states and formulae are straightforward, they don’t take a
huge amount of scrutiny. I’m tempted to say that if you cant
understand a salt state at first sight it’s not well written.

I think this depends on the referent of “you.” This certainly applies to folks like you and Marek, but I doubt it would apply to non-technical users, probably not even the average Qubes user. Nonetheless, a reason for hope.

[restive]

Hey, I’m a new user, and I have to say I’m blown away by Qubes, it’s a lot more user friendly than I expected it to be given the level of security that is possible with this system.

Regarding being inclusive of non technical users I would doubt non technical people install or setup Qubes by themselves, for a Journalist they likely have their IT folks involved in setting up the system. I wonder if part of the barrier to entry for some non technical users is actually budgeting engineer time to install and configure the OS. I don’t know if they typically hire outside contractors who specialise in Qubes installations or if they usually ask their general IT folks to install it, but if it is the general IT folks doing a small number of installs then even though it’s a technical user the UI and time commitment is still rather important.

Even as a technical user getting going with Qubes has been a serious time commitment, it’s not that I couldn’t get the operating system working just like any other linux distro in a similar amount of time but rather that if I want to benefit from the features of Qubes there is a lot of work involved in setting all of that up. Then if for some reason my yubikey breaks or my laptop dies I’m going to have to do all of that setup again, even if I have backups created that only covers the home directory.

Ideally there would be a list of developer maintained and (for the brave) a community supported repository of one click configurations like split-gpg with thunderbird or similar. This could actually increase security for users by reducing the barrier to entry for some of the more complex security configurations.

I dislike SaltStack in Qubes, learning SaltStack is quite a time commitment by itself and then you have to understand the differences that Qubes brings to the table. I’m not sure that Qubes leverages enough of the advantages of SaltStack for it to add real value. I think that becomes pretty obvious when you try to imagine what parameterising SaltStack would look like to avoid collisions with existing configuration. Square peg, round hole.

I wonder if a better arrangement might be something custom, something similar to docker compose that just defines the vm definition, relationships to other vms in the compose file and the bash script to run inside each vm on init?

[unman]

We are wildly off topic and I suspect that many people who might have
contributed now wont.

Pulling it back to the main topic - It wouldn’t be unreasonable to prompt
users on package install for certain information. E.g a VPN package
prompts as to whether all or specific qubes should use the VPN.

On interdependencies, look at the GitHub discussions of Qube flavors. No
reason why there shouldn’t be metapackages for complete out of the box
customisation.
Most repositories or package managers already have collections of some
sort. The description should explain what the Qubes system will look
like after install.

This discussion could be implementation agnostic - it could be bundles of
bash scripts.

[deeplow]

@unman

We are wildly off topic and I suspect that many people who might have
contributed now wont.

Just moved 17 posts onto a discussion which I called SaltStack Challenges and Experiences. Feel free to continue here on topic or the other thread for more staltstack experiences.

[gonzalo-bulnes]

Quoting myself for context:

(This is probably not the thread to answer that specific question, but I think the example may provide some scope for the use of Salt pillar.)

Following the suggestion to re-center the conversation:

I would add [to @unman’s questions]:

• How to provide guidance about how inter-dependent the packages (whatever they are) should be.

It seems to me that it makes sense that some packages build on each other, but also I think that such a repository would have less value if installing one package required to install a large number of them. (original)

@unman

We are wildly off topic and I suspect that many people who might have contributed now wont.

I certainly didn’t think of demotivating people to join the conversation by addressing some Salt-specific questions I found legitimate from the previous posts, I hope my intention at least was clear. Apologies if I did, I’ll back off.

[unman]

There’s a major hurdle in distributing salt packages using an rpm -
everyone says that you cant get interaction using an rpm.
This isn’t strictly true - you can hack the %post section to do some
interaction, and this works at the command line.
But since part of the aim here is to move away from command line use,
that’s not the solution.

I’ve been playing around with installing a simple script, and triggering
it in %post - the script would prompt (e.g):

Which qubes do you want to use the multimedia-player?
All - Some*select from drop down?* - I'll configure myself

and then salt using the newly installed formula accordingly - anyone have
thoughts on this?

On another front I’ve been hacking apper to only provide information
from specific repositories (to avoid offering options to install other
stuff in dom0), and to work properly with qubes-dom0-update.
That could be neat solution rather than building a Qubes specific
solution from scratch.