Following some recent issues on GitHub, as well as past discussions here, I am starting to question the whole concept of using a monolithic OS (like Linux) for the purpose of Qubes OS which aims to compartmentalize everything, thus creating a distributed computing environment.
The problems I see:
In Qubes OS, we would ideally like small entities working isolated from each other, having only what is necessary for their functioning. However, Linux is not made to be split apart and creating many minimized app-specific templates, each to be used with a subset of qubes is very inefficient.
Distros used for templates have their own specifics and goals different from those of Qubes OS (e.g. Debian - starting services upon installation by default and we don’t want to run software in templates). This perpetuates a long-term issue of having to either conform strictly to the upstream (with its potentially conflicting goals, potentially affecting the user), or following closely every upstream change for essential packages, then fine-tuning them for the purpose of Qubes OS. In both cases, this is additional work - either for the (aware) user, or for the devs.
Resolving any of these two big issues does not resolve the other. So, to my mind, this is a conceptual problem, not a procedural one.
Along these lines, I wonder if choosing a different basis is more suitable architecturally to Qubes OS. Perhaps a system which is designed to allow easy separation into independent parts that don’t duplicate each other?
Has there been any actual consideration/discussion of distributed operating systems as a more suitable concept for Qubes OS? Or what do you think?
That’s how all desktop software are made, it’s not really specific to Qubes OS. Server software are meant to be split in different parts like database, computing, storage, without various level of access and authorizations, but it’s specific and complicated. Not really practical for desktop usage for 99.9% of users.
In Qubes OS, if you want to run app1 in VM1 and app2 in VM2, your options are:
(easy) Install app1 and app2 in the same template, then any VM based on it can run them - a monolithic system.
(somewhat more secure) Clone a minimal template. In Template1, install app1. In Template2, install app2. Then have VM1 and VM2 based on corresponding template. This means 2, still monolithic, systems.
It doesn’t matter if the apps are desktop or server. The options are the same.
In a distributed system, which I envision, it should be possible to have:
a single minimal template (root.img)
each app installed its own compartment (app1.img, app2.img, …)
flexible AppVMs which use, e.g:
root.img + app1.img, or
root.img + app2.img, or
root.img + app5.img + app7.img
No duplication, no management overkill (20+ task-focused mini-templates with small differences between each other is not really minimalistic), centralized updates (no need for ng-cache and similar). If appX receives an update, other app images don’t need to be updated. Then practically everything is restricted to only what is actually necessary for running the specific AppVM and no app will be aware of the others. The root.img will run a limited set of tasks related to orchestrating this work.
Currently, disposables can combine one root.img and one private.img. That is very limited and does not solve the duplication problem. It also has a different goal.
I don’t know if the right term for this is distributed OS, but, considering the Wikipedia article, it seems quite close.
What you mean is that an AppVM can’t compose templates, it only has a single template.
This would be actually hard to stack templates on top of each other without having a hierarchy (let’s say layer 1 requires this exact version of fedora-38 template), this is what Docker is using when creating containers, it works well but everytime you change something in a layer you need to rebuild all other layers using it.
There is no simple method to compose/stack programs like this, at least not something as bullet proof as the current template system.
A simpler solution would be to do as @fslover suggests and install software locally using either flatpak or Nix.
To my mind, installing apps in an AppVM is against the security principles and the design of Qubes OS. I am not suggesting that. appX.img is like a template (not modifiable from qubes using it), i.e. not an AppVM.
How does it goes against the design principles (the most imporant of which is compartmentalization)?
In contrast to having software in template only, by installing it in an AppVM you allow that software to be modified persistently inside the AppVM. If one does that, Qubes OS is simply not necessary.
What you mean is that an AppVM can’t compose templates, it only has a single template.
There is no simple method to compose/stack programs like this, at least not something as bullet proof as the current template system.
I know but such vertical structure is necessary exactly because Linux is made to work as one whole, not as independent parts which can be combined with each other and updated independently.
It would be better if we had independent small atomic pieces which get necessary resources based on policy. For example (simplified), a basic mail qube needs:
file system to read/write
network stack providing POP/IMAP/SMTP
text editor (in fact, even that can be a separate qube itself)
It does not need audio or video player, USB functionality or gcc. However, following the current model, if one installs firefox, it becomes available to all qubes using the same template. This is just one example.
A simpler solution would be to do as @fslover suggests and install software locally using either flatpak or Nix.
It is not really a solution but rather a workaround of the current system. It would not be necessary, if it was possible to decouple completely programs (readonly) and data (read-write). I do it only for e.g. bash scripts written by me personally and even then I take care not to expose the qube too much.
It seems to me safer to compile from source some software, configuring it for a custom dir tree (e.g. /opt/somesoftware) without shared libraries, i.e. “portable”. Then put the build in the template’s /opt. The problem is that we can’t have opt.img separate from root.img, so we can “attach” it to an AppVM selectively, so the only workaround is to bind-dir /opt/somesoftware which. However, with default passwordless root that makes it writable from inside the AppVM. That’s why I see the current architecture as problematic and sub-optimal.
AppImage is a method to package a Linux program as a single executable file.
While a static binary is a single file containing all libraries, an AppImage is more complex and bundle extra binaries or librairies, but in a self contained file.
I know what is static linking. I didn’t know about AppImage. Thanks for explaining. I have spent some time researching these off-distro packagers after your reply.
So, to your question:
You could have a read-only NFS server full of AppImage binaries. What would be wrong with that?
IIUC, these binaries are downloaded from an external repository, i.e. from the distrusted infrastructure. This means we need to trust one additional entity.
Can it be trusted? On what basis? For AppImage in particular, I couldn’t find a single paragraph in the documentation discussing security.
Static linking is a double-edged sword - it kind of “freezes” the package, thus preventing both beneficial and undesirable (bugs/vulnerabilities) updates from influencing it. Gentoos seems to dislike it much.
You ask what is wrong with that. I actually wonder what is right with it (and how it solves any of the problems from the OP). Even the convenience is questionable, if one needs to manage an NFS server with such items. Additionally, it is usable only if an AppImage for the particular software actually exists.
Thanks for sharing your thought, I have been wondering lately if LXC / LXD could be an interesting component in Qubesos.
Using LXC ?
My 5 cent in the bucket : something not to be thought as a replacement for Xen, as LXC doesn’t provide such a strong dom0 insulation and cannot insure a “secured root” layer to my understanding.
This would be more a component to compartmentalize Apps in userland : think of your set of apps for personal, for work and so on which are living in the same AppVM. For this purpose, I am currently using snap.
Or to securely share os-level libs (if at all possible) : A template loaded with most of the application you need (the one you trust) could be the base for many AppVMs. And you would need other templates only for the most untrusted applications.
I don’t have a deep understanding of the system architecture so the examples are probably wrong. The main thing is about the concept of sharing the more resources possible instead of duplicating.
Using Salt ?
Another thing I am thinking of, that could ease the management of several templates depending of each other as layers, is trying to leverage salt to create (recreate) the whole set of templates I need from configuration files. If this is indeed possible, it won’t do anything in reducing replication, but salt could be used to describe the templates desired state and use inheritance (if possible in salt) to keep configuration dry.
The goal is not to make management of multiple templates easier. The goal is not to need more than one. The very necessity to have many is a proof of architectural inefficiency.
LXC
It seems to me dangerous to rely on less safe virtualization technology, especially considering the default passwordless root concept.
Or to securely share os-level libs (if at all possible) : A template loaded with most of the application you need (the one you trust) could be the base for many AppVMs. And you would need other templates only for the most untrusted applications.
I don’t know how to trust an app and the shared libraries it uses separately.
From what the OP was originally asking, based on my understanding, a micro kernel based approach may be applicable. This is where the kernel is the bare minimum instead of a monolithic kernel like Linux uses. All the operating components end up being services that can be added or removed independently.
A project that I previously followed that used this method, is GNU Hurd. However I have not looked at it in literally decades. Visiting the project site, it looks like their web design has yet to reach the 21st century, but the project appears to still be alive. They even discuss running Hurd under Xen.
If the micro kernel route is too low level, then perhaps a Qubes specific micro distro that just includes the kernel and not much else, would be more appropriate. However that would bring other challenges around software package support (E.g most software expects a popular distro, Debian/Fedora). A project of Qube’s size may not have the capacity to roll their own distro.
There may be existing source code based distro’s that would cover most of the OP’s original points, and be able to bring along their own package management systems to Qubes.