I’ve been using both, Qubes OS and Sculpt lately, and have been quite satisfied with both of them in many ways. However, there are a number of drawbacks in each that could be overcome if their strengths were combined.
Regarding Qubes OS, some of the biggest negative aspects are the large codebase in Xen and the complexity of Dom0, the need for virtualization with the associated overhead of running kernel and underlying operating systems in each VM (to perhaps run a single application), the problem of managing memory efficiently due to the need to rely on VMs to indicate memory pressure and prioritize somehow who gets what, the inefficient use of disk cache across VMs, etc. None of these are showstoppers and can be worked around (add memory, buy a faster CPU) or just taken as a (blue) pill you need to swallow as a counterbalance to all the significant benefits that you get with Qubes OS.
Scult OS (based on the Genode OS Framework) on the other hand, provides a capabilities based system with hardware isolation using memory segmentation/sandboxing, managed by a microkernel, with the ability to easily port linux applications and/or run VMs (it can even run a port of Virtualbox or Nova VMMs for easy execution of native operating systems). Sculpt OS is lien, fast and secure. The disadvantages are a small development community, relatively slow progress and a management/configuration system that is still quite clunky.
Has anyone considered a possible Qubes OS port that could leverage the underpinnings of Genode to replace Xen and eliminate the need for virtualization just to provide isolation? I believe that it could create a better/faster Qubes OS but, of course, at a price of a pretty significant porting effort.
Generally, this is a very good idea, and we have considered it in the past several times. Genode OS Framework has a very clean and though out design and also has excellent documentation.
But as you said, the effort for porting is very significant. There are several paths we could achieve that, but all are of the scale of “several man-years of work”. I hope we will have enough resources for that, one day.
BTW, one of the key differentiating factor of Qubes OS versus various microkernel-based systems, is ability to run unmodified applications (Linux, Windows). It comes at a big price in terms of resources, of course, but still that’s something we are want to keep. That means, even under Genode OS Framework, we will likely still use virtual machines, even if for less parts than we use them right now. On the other hand, you can run much smaller VMs for specific tasks under Xen - namely, you can use unikernels. And there are a few implemented specifically for Qubes OS (look for qubes-mirage-firewall, or qubes-mirage-ssh-agent). This is a way to bring some values from microkernel-based systems to Qubes OS as it exists right now.
@marmarek Thanks for the fast reponse! Unikernels/exokernels/library are indeed a great idea to reduce the overhead and keep the binary compatibility so I’ll surely look at mirage.
Along the same lines of thought, would usermode linux provide adequate isolation and reduced overhead? I haven’t looked at it in years, but had some good experience with it back in the day. If this works, it would pretty significantly reduce the number of context changes (userspace to kernelspace) and may render a nice speedup and memory footprint reduction. Of course, it could change the current paradigm from one security domain per DomU to several security domains in a single DomU (each running their own usermode linux in their own application memory space).
The only other aspect that may be useful to consider from Genode is the explicit (intention based) capability model, where specific capabilities are offered from the host to a specific application/vm and only these capabilities are available (rather than a general hypervisor interface) in order to reduce the attack vector footprint. It’s possible that SELinux in Qubes already offers something like this for Dom0, but at a pretty significant complexity cost (SELinux is a collection of poorly implemented great ideas, IMO ).
Last, but not least, I have seen what some of the NixOS folks are doing to bring their declarative configuration system and immutable stores into Qubes OS, and it seems promising. Spectrum also looks interesting since it’s taking the concepts behind Qubes and merging them with Nix package manager, but with only one core developer (the founder of the project) for the last couple of years, I’m not sure if it will eventually become ready for mainstream use.
Anyway, thanks to everyone in the Qubes OS development team for the great work!
I’m afraid usermode linux doesn’t provide isolation level we require. It basically relies on Linux kernel (including its fat syscall ABI) for isolation, and that’s exactly what we try to avoid (otherwise, we’d use just containers instead of VMs). Some of it can probably be hardened by seccomp filters and/or SELinux (I guess, a usermode linux process can be confined tighter than arbitrary user application, as many of the APIs will be provided internally), but that’s still a huge change in provided guarantees.
Yes, that’s important benefit. On a very high level, it is kind of similar to what we currently do in Qubes - VM has access only to resources it needs or were explicitly assigned (explicit assigning devices, qrexec policy framework etc). The API of the hypervisor is not that big (especially after we stopped using PV in most places). The most significant difference is the nested structure in Genode - this allows using using the same capabilities model not only on a high level (“VM”), but also to isolate elements within such components (in Xen, each VM must provide own internal isolation mechanisms, if they want to use some). That obviously has a cost on the compatibility side, but there are several elements that are Qubes-specific anyway, and could easily use such mechanisms.
@marmarek Regarding usermode linux, you have a very good point there. On the hierarchical capabilities model, yes, that would be quite useful, especially if that could be extended to individual applications within VMs, and even within Dom0.
Going back to your reference to Mirage, is it currently at a point where it could be reasonably tested within Qubes? I’m not sure how beneficial would be to just move the sys-firewall qube to it, but it could help if other areas can be moved too (sys-net comes to mind, since it doesn’t allow for memory balancing, so allocated memory is locked into it).
The RAM usage difference is significant - ~64MB vs >400MB.
That’s actually one of the VMs that’s hardest to move to unikernel. The reason is simple: you need there drivers for all kind of network devices users might have. Most unikernels and similar solutions have rather limited support here. This BTW applies to Genode too…
@marmarek Good point about the need for device drivers. Yeah, hardware compatibility is an issue so those drivers are needed there. I’ll look at the document that you pointed out and maybe take mirage-firewall for a ride. The beauty of Qubes is that you can test different things without much fear of breaking things
Your post and the unique properties of Sculpt/Genode made me interested to try it out, however I have to say even after using Qubes for more than 5 years I find Sculpt to be way more difficult to use and configure than Qubes, with some guides being outdated and becoming inaccurate I find myself having too little knowledge to be able to really use it. It is very interesting though, I hope a future version will be more user friendly for non-experts, and maybe they could offer a secondary download with more things preinstalled and configured for ease of use.
You are right. It seems that with a small development team, the Genode project people are spread thin, trying to port it to other platforms (Pine phone, for example) and, at the same time, developing new capabilities. Architectural documentation is excellent, but good how-to guides are scarce and limited in scope.
If you know what you are doing and have a reasonable internet connection, you can go from live USB Genode/Sculpt to full system in about 20 minutes but it’s not straightforward. You will need to know which depots you need to use (Genodelabs doesn’t have, for example, the native Falkon browser included) and figure out how to wire each component to the correct capabilities and services. Once you have a working system, which can be done, as I said above it about 20 minutes, you end up with a reasonably secure operating system, a Debian VM, a native web browser and basic support for your hardware, but you don’t have persistence yet. Yes, you downloaded the packages into your expanded USB filesystem so they are local, but the configuration that you created through the configuration GUI is volatile and stored only in memory, so you need to copy the config files (depot, network, etc.) to your USB filesystem by hand using the console (and even create the correct directory number based on the Genode version, for example, 22.04), or you would be back to a blank slate after your reboot.
As you said, they need some help to get Genode/Sculpt to a state where more people can use it for day to day activities. What I was proposing above was perhaps to take the best ideas from it to enhance Qubes (for example, to eliminate the Xen/Domain0 dependency, which bothers me and a few other people in this community). Ideally, having a far smaller codebase in the trusted components of Qubes would make it a lot more resilient to security defects in the future.
Yes, I found that Falkon Browser and Debian VM for example are in other depots. But how to wire each component is not explained for every software. I found a guide for VirtualBox, which does explain, but the current version in the depot has more components to configure, and some options ‘fall off’ the screen because the display resolution is too low, and I can’t move the GUI. I found a way to change the display resolution on a mailing list, but that didn’t work for me. I also though I had Falkon Browser configured correctly, but the application window just doesn’t show up, even though others like shell and the nano3d demo do succesfully, and I connected the GUI to the themed WM. And sometimes the depot just won’t install sofware I select or the GUI freezes completely.
And then there’s creating the persistence yes, too complicated for me
Anyway, I’ll give a future release another try sometimes.
But combining Qubes with Genode does indeed sound great, it is a shame that it requires so much work. I hope that Qubes OS will get more sponsors in the future. I saw Mullvad upped their donation to $100K this year, that is great.
).