Spectrum OS discussion

General Discussion
1

Hi all,

I recently learned of Spectrum OS from this tweet:

https://twitter.com/pavolrusnak/status/1327943049842921472

Quick summary from the developer, Alyssa Ross:

a NixOS distribution focused around security through compartmentalisation in the style of Qubes OS, but with the diversity of hardware support and ease of maintenance afforded by the Linux kernel and Nix.

On the motivation page, there’s a discussion of Qubes:

Existing implementations of security by compartmentalization

Qubes OS

Qubes OS is a distribution of the Xen hypervisor that isolates IO and user applications inside their own dedicated virtual machines. Many people interested in secure computing are aware of Qubes, however they are often hampered by usability issues:

  • Hardware compatibility is extremely limited. People often have to buy a new computer just to use Qubes, and even then it can be a struggle to set up.

  • People are reluctant to use Xen on their computer for power management etc. reasons.

  • VMs are heavy, and there is no isolation between applications in the same domain (VM).

  • GUI applications are buggy, command line tools are mostly undocumented.

  • Maintaining many different TemplateVMs with persistent state is difficult. (Qubes can use Salt to mitigate this.)

It is important to note, however, that the Qubes developers have created utilities for using compartmentalized environments that could be very useful to other implementations. For example, Qubes allows clipboard data to be safely shared between isolated environments with explicit user action on both ends, and Qubes Split GPG allows one environment to perform operations using a GPG key stored in another environment, with permission granted on a per-operation basis.

The design page goes into more detail:

https://spectrum-os.org/design.html

I thought this might make make for an interesting discussion topic, and I’m curious to hear what you all think.

6 Likes
2

From a layman’s perspective my first impression is ambivalent especially with reference to the list highlighting the alleged downsides of Qubes.

I was immediately reminded of crowd-funding campaigns and the comparisons with check boxes at the end of the campaign page. Competition is usually made to look bad/weak and the categories are somewhat dubious. This categories are often extremely simplified and while mostly containing a grain of truth or more it is not seldom only half the truth. There are two sides to every coin.

Regarding this list:

  • Hardware compatibility is limited but measured by the relatively small number of users the HCL (hardware compatibility list) is quite substantial and growing with the user base. There are a few older models that are cheap, easy to come by and reliable. It is also clear that there are a few must-haves for example for virtualization.

  • who are these people? I guess IT professionals? I don’t know if most people know a lot about the differences between Xen or KVM in order to make an educated statement weighing the pros and cons (?)

  • again, there are heavy VM and there are alternatives like mirage, minimal-templates etc… There are solutions like firejail or containers for additional isolation, also adjusting the policies to make use of disposable VM is helpful (can be tricky - pay heed to the warning signs)

  • yes and no. In my mind the documentation is one of the strengths of Qubes so far. I really think the documentation is excellent already and if more people read it more closely I guess half of the questions wouldn’t be necessary.
    Of course, there is always room for improvement but this project is a work in progress that is constantly being developed and everyone can take part in improving things like helping with the documentation or bringing propositions to the table.
    “mostly undocumented” a few examples would have been nice, because apart from the bleeding edge stuff that is being developed and not ready (e.g. gui-vm, or in general Q4.1) yet I think Qubes-specific commands are explained quite well whereas a lot of commands that lack documentation are Linux-specific and can be looked up in the respective Wikis.

  • I agree, salt needs some time and training (at least for me :wink: - I am trying to learn using it on an experimental install that is for educational purposes only)

My point is, you can look at the points made from another perspective and it might look a bit more positive.
That said, it sounds like an interesting project with a huge ambition and it will be interesting to see if the developers can keep up with their own goals while simultaneously keeping it simple and well documented.
(I am careful with predictions because IT is not my area of expertise but I am a bit skeptical because Qubes is being developed for quite some time now and disregarding the manpower and funding of spectrum this is a big plan that will need a lot of time)

5 Likes
3

In my mind the documentation is one of the strengths of Qubes so far. I really think the documentation is excellent already and if more people read it more closely I guess half of the questions wouldnt be necessary.

Absolutely. Another thing that many users don’t seem to “get” is that
the qubes they are working with are pretty standard distributions.
So that the problems they encounter can often be solved by looking in
those distributions. This should be highlighted somewhere, or flashed
on to the screen every 5 minutes or something.

I don’t like Forums, I don’t like Reddit - one reason is that users seem
to use them as the first point of call instead of trying to read and
solve problems for themselves.
Also they seem to encourage the use of images: a huge down side for me.

I agree, salt needs some time and training (at least for me :wink: - I am trying to learn using it on an experimental install that is for educational purposes only)

You might like to look at
notes/salt at master · unman/notes · GitHub.
They are notes from some training, that move from basic commands up to
more complicated configurations. You may find them helpful.

5 Likes
4

A Nix(OS) and Qubes OS newbie here. :smiley:

NixOS as the base distribution is interesting but the AppArmor and other LSMs and audit subsystem supports are in progress so I would hope there will be some progress boost in this area thanks to Spectrum OS. Also, I am looking forward to realizing /nix/store as a cheap way to share the system components between domains but it may be a bit tricky to do it safely.

I am not sure Qubes OS’s hardware requirements are extreme or not — sure, my cute X200 is sadly out of support, but the VT-x/EPT/VT-d-equivalent virtualization capabilities are very common nowadays as far as I know. The real pitfalls would be that some system firmware disables VT-d by default and powerful laptop PCs have dGPU, in which the letter case seems very annoying for some people.

Xen vs. KVM: my understanding is there are some trade-offs and indeed the performance of KVM is promising, but Xen’s recent progress on further isolation and dom0 shrinking is excellent too. I hope Qubes OS and Spectrum OS will be good competitors in this area.

Xorg vs. Wayland: yeah, Wayland seems neat but implementing and stabilizing the user experiences of Qubes OS on top of Wayland will need non-trivial human and time resources, I guess.

That said, Qubes OS might have some use-cases of Nix, both the package manager and the distribution. I would become a heavy user of NixOS TemplateVM for sure. If there is a mechanism to share /nix/store between VMs without making all of them accessible from all AppVMs, it would be a killer feature IMHO.

2 Likes
5

There is also Bottlerocket, an open source Linux distribution built to run containers:

Bottlerocket is optimized for running containers with high security isolation. The host OS is extremely minimal, it does not come with bash, an interpreter, ssh, or anything beyond the system basics needed to run containers. In fact it uses an immutable root filesystem. You aren’t intended to run or install things directly on the host at all.Bottlerocket is optimized for running containers with high security isolation.

2 Likes
6

I don’t want to belittle that project, but I’ve written a fair number
of pitch documents, and an essential part is to belittle competitive
technologies, so I don’t take those comments too seriously.
The grant assessors simply wont have fact checked those claims.
Also, there is scope for Qubes on KVM (It’s been in the specifications
for some time) and there has been some movement toward it.
That said, it looks like an interesting project, which may have some
feed in to Qubes.

NixOS template? I had one for a client some time back. I’ll see if I can
dust it off and release it.

5 Likes
7

Thanks a lot! These notes look like to be very useful indeed. This applies to the other notes as well. Very nice collection of information!

8

Whoever objects on this, cannot be taken serious by me. The fact speaks about hardware, not about Qubes, so this story stops at their first remark for me.

9

Actually, not everyone has funds to buy new hardware when they want to switch to Qubes. Due to that, the hardware compatibility plays an important role, whether it’s Qubes fault or not.

2 Likes
10

I agree partially but first, one has to define his/her priorities. It isn’t that you throw your hardware to garbage and have to buy a new one.
So, it hasn’t to be expensive: sell your fancy hardware, buy used/refurbished one and you can get extra money saved.
Or, sell whatever laptop you have and add a bit more to buy Qubes HCL one.

I think I could bet that HCL is minor reason to the ones who decide not to use Qubes. Majority who give up find it “not user friendly”. A poll could prove or disregard this.

I don’t like to write about my personal stuff, but my laptop is way, way too old and it smoothly runs 4.1rc2 with 12 VMs at the moment, and it is only because I’m emerging 700 packages of gentoo right now and it needs 8GB RAM for that. Usually I’m running 15-18 VMs concurrently. Twenty years ago I could never imagine I could use hardware for more than 3-5 years.

And, as I said, this is “… for me”. No incidence could keep me away from Qubes, and for sure I will use it as long as possible after it gets abandoned (which I doubt).

1 Like
11

Comment on the very same tweet

ТМР
@tmrcz

·

Nov 15, 2020

I didnt want to use Qubes because it always seemed to be a bit too heavy and prescriptive for me. I am that kind of person who likes to have control over everything, use specific tools, window manager, etc. Qubes didnt even boot in my VMware so had not much real trialing yet.

This is what I am writing about.

And what is their “HCL”. if I understand English well enough:

I would like Spectrum to additionally have first class support for at least ppc64le.
…
Unfortunately, preliminary research has shown that x86_64 virtualization on POWER9 is not currently sufficiently performant for this to be feasible.

12

There is a recent video demo of VMs in Spectrum:

3 Likes
13

Further to:

On their website, they state:

In Qubes, user data in each VM is stored in its own virtual block device. This works fine when multiple applications run in a single virtual machine, but would be unmanageable in Spectrum’s VM-per-application model.

(emphasis mine)

Given that most of my qubes are dedicated to a specific application, (as many users’ here are), this is disingenuous. This is on their hompeage, it’s not hidden away in a funding-proposal, which does not make me think too highly of them.

If they instead said, ‘this would be unnecessarily complex in our per-application isolation model’, that would be correct - but they did not say this. The term ‘unmanagable’ is effectively a direct-blow at Qubes.

Otherwise, I welcome the project.
KVM will bring compatability to more devices. I like what NixOS is doing and I like their application-security model.
Although I may use spectrum-os when it releases, I shan’t be switching from Qubes. Of the information I’ve seen their security model is not as good as Qubes’s, I have some concern already with the measures they proposed ‘could improve performance in the future’.

I do have some questions regarding crosvm, I’m aware of it but unfamiliar. (Should I make a separate thread for this?).
Entirely upon principle, replacing QEMU with virt devices (like virtio for xen) is good.
I’m aware chromebook’s/chrome-os security is rightly praised in many areas, however, given that:

crosvm is a virtual machine monitor (VMM) based on Linux’s KVM hypervisor, with a focus on simplicity, security, and speed. crosvm is intended to run Linux guests, originally as a security boundary for running native applications on the Chrome OS platform. Compared to QEMU, crosvm doesn’t emulate architectures or real hardware, instead concentrating on paravirtualized devices, such as the virtio standard.

crosvm is currently used to run Linux/Android guests on Chrome OS devices.

(emphasis mine)

I don’t imagine crosvm provides the equivalent level of security an abstracted virtio does.
I’d like to know your thoughts. How would the security compare in a hypothetical KVM+crosvm system vs xen w/virtio?

Edit:
Regardless, crosvm is a google-controlled project and AFAIK has little market share outside of google’s usage - which is not at all positive for an opensource project.

14

I agree that ‘unmanagable’ is a strong term, and somewhat disingenuous. However there are big differences between the projects in my understanding. Qubes requires a different mindset to compartmentilize your digital life into different domains, that may go as far as running one application per VM. (Which can be more secure, but that depends on how you use it. If for example you have a Thunderbird VM with your work email account, private email account and some unimportant email accounts for registering less important / throwaway accounts for example, then an exploit in an email for any of those accounts can compromise all of them.) While Qubes makes managing VM’s much easier with application windows from multiple VM’s on one desktop, TemplateVM’s for updating, less disk usage etc, a lot of tools for managing the VM’s, it is still a system a lot more complex than a standard Linux distro. You have to manage your TemplateVM’s, standalone VM’s, deal with the fact that all VM’s have their separate filesystem and send files between VM’s. Is it a lot easier than just using Virtualbox on Linux with a lot of VM’s? Yes, definitely, and also more secure. This is why I love Qubes. Spectrum on the other hand seems to aim more toward noob users. With all applications running in their own VM, shared files between VM’s. not having to deal with updating seperate templates etc. To my understanding the end goal of Spectrum is that a noob user can use Spectrum securely without even being aware that he is using separate VM’s. It will not be as secure as Qubes, but a nice middle ground between usability of a standard OS and the security of Qubes .
In that view, each VM/Application having it’s own filesystem IS unmanagable for noob users, who are sometimes struggling to find in which folder they saved a file in just 1 filesystem, having multiple would already be really hard for them, let alone 10 or more for example.

To my knowledge, Spectrum has largely switched to Cloud Hypervisor, which is largely based on crosvm, but not a google-controlled project.

15

Well, there might be a situation I’d look for an alternative to Qubes…

16

Hello @unman is there any possibility to share your NixOS template with us, please?
Or any instruction on how to make one?

17

found something related to nixos:

1 Like