I very much like what Marek wrote earlier this year:
[…] at the point when we’ll be minimizing dom0, more likely option
would be something capable of building light system images, for
example Yocto."
" Back to your assertion" - don’t know which assertion you refer to.
Since Centos Stream is a rolling distribution, (rolling-minor, ha)
it cant be a serious candidate for dom0, regardless of its weight.
I doubt @marmarek read Yocto’s security page:
Yocto Project does not have a Security team … there is some research and proof of concept work occurring with some tools but its struggling due to lack of people/resources.
And since this thread is about RPM distros … what do you think about SilverBlue/CoreOS or SuSE MicroOS? SilverBlue and CoreOS leverage git to provide immutability, de-duplication, and rollbacks but leaves us the option to use DNF, which would make the transition easier. I know those aren’t the only alternatives, but be wary of marketing fluff:
…
This one:
I’m skeptical that it will be any less stable than Fedora or what we can get for free from volunteers reverse engineering code drops from RHEL, like CentOS used to do. It also means security fixes take 24-72+ hours to hit the update servers. Seriously, go read the CentOS Blog about the new integrated release engineering process:
Whatever work that is under NDA or embargo, that will eventually get reverse engineered by Facebook (or whomever) and pushed onto CentOS Stream.
Do you have any input on letting users choose between CentOS Stream and a RHEL clone like Oracle or Rocky Linux?
Fedora Silverblue, CentOS Stream, or a RHEL clone all seem like acceptable options to me. Silverblue has the potential to reduce breakage, but I think that would have to wait until the GUI is out of dom0. The primary objection to RHEL was that some packages don’t work and it is stuck on old software. But with a major release being cut every three years from Fedora via CI, it seems like a viable option.
Or at least @DemiMarie expressed a preference for CentOS:
I think that CentOS would be a good choice for dom0, as it can have a recent kernel, yet is extremely stable and has signed packages and signed metadata.
And @fepitre is the guy who handles CentOS stuff and mentioned CentOS Stream:
dom0 is in a rather weird position. It needs to be stable, so that changes don’t catch the Qubes core team by surprise, but it needs the latest hardware support and installer. I doubt there are any existing distros that are good fits for us. A derivative of CentOS 8 is the best option I know of, but a fully custom distro would be ideal.
I agree. I’m seeing that CentOS 8 Stream has moved forward a lot. I could redo some tests but well, that would deserves to be a specific task and @marmarek would need to arbitrate this.
Please stick to constructive feedback.
The Problem with dnf Always come back
I think its better leave Fedora yo a more Security distro or Os for dom0
I doubt @marmarek read Yocto’s security
page
Hm, that kind of misses the point. Let me try to be more verbose:
What does dom0 do currently?
- hosts the GUI (XFCE)
- hosts the audio subsystem (Pulse)
- Kernel
- XEN
- other hardware is already isolated in sys-net and sys-usb
I think what makes most posters so passionate about which distro runs in
dom0 is their wish to have a different or more up-to-date GUI
environment. That goes away with the GUI VM.
What’s left? What does dom0 do in future?
- Kernel
- XEN
- Qubes specific binaries
This is what I understand when I read “minimizing dom0”.
→ There is no longer a need for a distro. ←
Maybe I am wrong, but that is what I am expecting to happen and it looks
more secure to me than trusting any distro outside the Qubes project.
1 Like
Experience shipping production systems matters. Alpine was shipping ‘null’ as the default password for years.
A “distro” provides us with a critical mass of support, both in terms of eyeballs on the code and support from hardware vendors. Community distros haven’t poured millions of dollars into testing and delivery infrastructure.
And aside from being shiny, what does Yocto or Alpine do that CoreOS doesn’t?
And aside from being shiny, what does Yocto or Alpine do that CoreOS doesn’t?
Again, I am not an expert. But the point as I understand it is the
reverse of your question. The goal would be a dom0 that is extremely
minimal, which means: reduced attach surface.
Currently we need a distro in dom0 because the GUI lives in dom0. Once
that is no longer the case, there should be no reason left for any user
to interact with dom0 in any way. There shouldn’t be anything anybody
would want to install, configure or maintain in dom0. It should just be
the Kernel, XEN and Qubes daemons.
That’s what CoreOS is all about: reducing the base system to the maximum extent possible. Red Hat acquihired the leading container oriented distro and brought a bunch of process enhancements (like better release QA and back-porting) to make it more suitable for use in production.
The idea that Qubes doesn’t need a base distro is silly: Linux is just a kernel and it doesn’t make any internal stability guarantees. A distro is a coalescence of disparate upstream sources, which is a lot of work. Anyone can roll-their-own Linux distro badly, but its much easier to draft off of Debian or Red Hat.
Yocto and Alpine advertising themselves as minimal is less honest than the sticker price on a new car. 2.5 megabytes won’t get you running on Xen or boot a laptop. After you shove in everything required to actually do anything Yocto and Alpine are the same size as everyone else. The main difference is that they don’t have nearly as many resources to make sure it all works correctly.
Even if dom0 doesn’t require a GUI, we need a GUI somewhere and it’s more efficient if we use the same base system in dom0 as we do in the virtual machines. At a minimum, you get space savings by de-duplicating what’s on disk. But compatibility is the biggest issue: we literally can’t use CentOS right now because the userland is not in sync with Fedora and no one else has bothered to backport everything. Which leaves us stuck with a lot of extra porting work.
Ideally, we will someday get to use SeL4 and run everything in WASI isolates. But Qubes is not a research project and it needs to run on real hardware.
it’s more efficient if we use the same base system in dom0 as we do
in the virtual machines.
How and why?
At a minimum, you get space savings by de-duplicating what’s on
disk.
Shared files between dom0 and a domU? I don’t think so.
But compatibility is the biggest issue: we literally can’t use
CentOS right now because the userland is not in sync with Fedora and
no one else has bothered to backport everything. Which leaves us
stuck with a lot of extra porting work.
I see. You want to use CentOS as domU and are frustrated by the amount
of work that needs to be done to get there. I can empathize with that.
That doesn’t make it a good or better choice than what we already have.
In addition having a rolling distro by definition would be the opposite
of what we need in dom0. There should only be very very tightly managed,
infrequent and necessary (security & bug fix) updates to dom0 after release.
I lived on a rolling distro before (SuSe Tumbleweed). It was fun but not
stable in any sense of the word.
/Sven
For the same reason that it’s more efficient to target a single execution environment in any other setting, be it browsers or specific versions of specific operating systems.
Like the Linux kernel or userland. You know, whatever is in the 130 megabytes of the Alpine Xen image ; )
No, I am not. I am talking about using the exact same version of hundreds of different pieces of software that makeup a distro. For example, there are at least 75 packages (see: [1, 2]) that Qubes uses in Fedora which are not available by default in RHEL.
What we have is Fedora, which is only stable for 1 year and porting between major releases is a resource drain on the project.
I will repeat this point one last time: you don’t have to use CentOS Stream. The project could use Rocky Linux or any other RHEL clone.
CentOS Stream is downstream of Fedora and represents a minor point release for RHEL. So no major breaking changes, just new features and bugfixes. I’ve had Fedora stop booting on some laptops because it will switch between kernels without a major version bump. So I would imagine that CentOS Stream would be more stable than Fedora.
@Sven I’m not finding any of your responses productive. You advocated for a minimalist distro, then switched to expressing dislike for CentOS Stream. I already addressed most of the concerns you raised, either in this thread or in the Github tickets. And you don’t discuss alternatives that are based on an RPM or RHEL adjacent distro.
I get the impression you do not like Red Hat, which is fine. But this thread is focusing on RPM based distros. Not because I am a Red Hat fanboy, but because it means we don’t have to replace all of our infrastructure at once. Some non-RHEL adjacent distro may be better for dom0. That’s fine, but please discuss that distro in another thread.
That being said, I’m not wasting any more time entertaining low effort comments. I will flag anymore off-topic comments as such.
Thank you!
It’s not “off-topic”, “low effort”, or “not constructive” to question
the assumptions underlying this thread or your assertion that:
CentOS Stream surfaced as a top choice for a few reasons
In fact, it seems that following those discussions is far more
productive than a narrow focus on “what RPM distro”, and productive in
the long run.
Identifying what needs to be in dom0, researching possible candidates,
and the work that would need to be done, that’s useful. This isn’t.
@Sven I’m not finding any of your responses productive.
I am sorry to hear that.
It was my intention to share my understanding, which is that the choice
of distro in dom0 has very little impact beyond hardware support now and
will matter even less in the future.
We have Fedora now for historical reasons. The project lead pointed out
that once dom0 is minimized, we might get rid of Fedora in dom0 and
switch to something minimal.
In this context I recognized that we perceived things very differently
and it might be interesting to have a conversation to uncover incorrect
assumptions.
I get the impression you do not like Red Hat, which is fine.
My preferences boil down to a specific look & feel and stability (no
updates except security patches). I couldn’t care less about the distro
or the packaging format in domU.
In dom0 I want no changes or user facing features at all unless they
significantly improve security.
this thread is focusing on RPM based distro not because I am a
Red Hat fanboy but because it means we don’t have to replace all of
our infrastructure at once. Some non-RHEL adjacent distro may be
better for dom0. That’s fine, but please discuss that distro in
another thread.
I understood that from the start and tried to reason with you why
arguing for a distro change in dom0 might not be the best use of your
attention.
I largely started this thread and the Alt RPM Distro in dom0 ticket as a brain dump after doing a bunch of research. I was willing to rehash discussion from Github because I thought it would be helpful to have the discussion all in one place. But now it feels as if we are retreading ground already covered in this thread.
The vast majority of stuff in a Linux kernel is for hardware support. What is the testing infrastructure like for most distros? I would be surprised if anyone outside of Debian and CentOS have a good hardware testing infrastructure.
Define minimal: you can play at the margins, like replacing some GNU userland stuff with Busybox or a smaller standard C library. But those alternatives compromise on features and aren’t going to be as well maintained. Since the application VMs will need the standard GNU userland and C library, we might as well just use the same distro in dom0 and dedupe the base image. At that point, it feels a lot like arguing about sports teams: it’s all the same code, just with different levels of integration and infrastructure.
What couldn’t we accomplish using a Fedora Spin, CentOS Variant, or even a Debian Derivative? SilverBlue/CoreOS makes this very easy. Then we can push Qubes specific software into the distribution’s continuous integration system, with all the benefits that provides 
.
Wanted to highlight two developments:
- Red Hat has expanded their free tier to 16 RHEL licenses for use in production; users unhappy with CentOS Stream could switch to RHEL for both dom0 and 15 appVMs without paying anything.
- Red Hat has a free for FOSS infrastructure program; Qubes developers shouldn’t need to worry about bumping up against the 16 cap of the aforementioned program.
I also realized that dom0 gets updated between ~1 and ~3.5 years (~1.9 years on average). If Red Hat sticks to their revamped 3 year release cycle, we wouldn’t be stuck on old kernels much longer than what we are used to. The CentOS Hyperscale SIG provides accelerated backports (slides) of specific items (like systemd) and is working on backporting LTS kernels as well (which are released ~yearly). So it should be feasible to keep dom0 on a fresher kernel in future.
I think this could be something interesting to post on the qubes-devel mailing list so it more easily reaches the developers (only a few see the forum).
If you feel like it’s gotten to the point of being useful, feel free to ping them 
. I’ve also started up this meta-thread.
1 Like
Some arguments for OpenSuse
I’d like to collect some Qubes-Dom0-specific arguments that would be in favor of OpenSuse.
Transactional Updates
The transactional-update feature could add a lot of benefit to Dom0, especially:
- Easy and stable file system snapshots,
- the read-only root filesystem can increase Dom0 security.
Good Stability
The Tumbleweed rolling release could be suitable for the testing release (currtently R4.1), while the Leap versions could be used for the stable release (currently R4.0).
I think OpenSuse Tumbleweed is the most recommendable rolling release distribution to daily drive as a desktop user (not sure when you have tried it the last time). They put quite some effort into making Tumbleweed as stable as a rolling release can be.
In combination with the filesystem snapshots provided by transactional-updates, Qubes’ testing release could be updated frequently without fear of breaking those installations permanently.
Good Xen support
OpenSuse is built to work with Xen. You can simply install it through their superb system administration tool YaST.
On other RPM-based distros I remember that Xen often requires some tinkering until it works properly.
Acknowledgements
Of course, there are a lot of big arguments against OpenSuse, too:
- Why considering a distro that has not even proven to be useful as a template?
- The
transactional-updatefeature might require some adjustments regarding packaging until it would work. - The effort that is put into making Tumbleweed as stable as it is, will not benefit Qubes tools without further effort.
1 Like
Thanks for a productive post @wind.gmbh!
AFAIK, they also have put in a lot of work towards reproducible builds. I certainly think they would be more suitable than a purely community backed effort like Arch.
I think I would agree with you in that it’s probably the most professionally run rolling release distro, but I
don’t think a rolling release is desirable. Of course, I don’t count CentOS Stream as a rolling release 
.
Jumping late in this discussion.
After having read the whole thread (awesome btw) and having followed GitHub discussions for a long while (debate of switching to Debian for a long time with PoC having happened), I would love to understand concise criticisms against SilverBlue being the next step moving forward for Qubes, both for Templates and dom0? @unman? @marmarek ? @adw
As for Templates and release upgrades (Q4.0->Q4.1 being the last good example of problematic and iterative patching (there is still problems upgrading from Q4.0 to Q4.1 as of today, reported on GitHub), I again believe that a rolling Qube release would be ideal, while problematic, while SilverBlue seem to be a good middle point instead of moving to Nix/Guix, or switching to debian (longer release cycles, far from actual Fedora, drivers availibility problems etc)…
So my short question would be: have somebody tries SilverBlue enough to give an technical opinion of what would stop dom0/fedora template to be based on it? Relevant issues or other discussions pointers are good enough!
1 Like
I would love to understand concise criticisms against SilverBlue being the next step moving forward for Qubes
SilverBlue would be very cool and there have been experiments that fully virtualize Flatpak apps. The problem is that Qubes doesn’t have the engineering resources to make the leap.
Even if they did, it’s not clear how the SilverBlue experiment will turn out. I think SilverBlue sees itself as the future of RHEL, but they are still busy building that future. RedHat might pivot and make major structural changes or drop the project entirely. Qubes would probably need to become a SIG within Fedora to influence engineering decisions. But even that wouldn’t guarantee they would be able to get the financial backers to put any effort into, for example, making Fedora or CentOS fully reproducible (because who would pay for a RHEL license if they could mathematically verify that their builds are identical).
TBH I think these discussions about alternative distros are moot unless someone advocating for a specific distro is willing to either put in the time and effort required to make it happen or pay someone else to do it.
I’d love to see SeL4 as the hypervisor, but just talking about how cool SeL4 is won’t make that happen any sooner… 