We can use Gitian to make the existing system reproducible.
CentOS
CentOS Stream surfaced as a top choice for a few reasons:
CentOS signs both packages and repo metadata.
The CentOS community is repositioning itself as the integration point for multiple RHEL derivatives.
Stream roughly correlates to latest minor release of RHEL.
CI pipeline Fedora -> Stream -> RHEL.
Not RHEL-beta: Red Hat, FB, AWS, Oracle, VMWare, SuSE, etc will still deliver some patches to their own customers before they are cherry picked for CentOS.
New major version every 3 years, support for 5 years.
Oracle provides a binary release based on newer kernelsâŚ
The biggest sticking point was that RHEL/CentOS has fewer packages than Fedora. The push to CentOS Stream is about streamlining the pipeline from Fedora to RHEL, so it should get easier to push Fedora packages we rely on downstream. I donât know much about their community, but AFAICT CentOS Special Interest Groups are independent from Red Hat. The Virtualization SIG was highlighted in various blogposts and is probably the best place to start.
However, how much of that could be solved feature gating development? I was able to find many of the packages from @fepitreâs coprs (epel-8-qubes, epel-8-python38) in EPEL and AppStreams (see 8.3.2011 or search via pkgs.org). Some vendors (like Salt) want you to use their repos directly. I was able to find even more (albiet older) packages on CentOSâs Pagure/Git hosting.
Silverblue
Once the GUI layer gets moved out of dom0, the dependency on OS packaging becomes much smaller. At that point Fedora Silverblue is likely the next best choice. I wonât recap explain all of Silverblue here, but it is a container focused immutable OS that uses Git to manage the OS layer. But we can still install and use DNF, so the transition would be easier than a purely functional distro like Nix.
My reading of Streanm differs from yours.
Centos Stream does not âroughly correlate[s] to latest
minor release of RHELâ - itâs intended to be always slightly ahead
of the latest release.
Centos Stream will be a development project for RHEL.
I donât think that dom0 should be based on anything but a rock steady,
slow moving distro.
+1 second that.
Always thought dom0 should have been Centos, since dom0 is really mostly a server, and does not run apps.
Centos stream will be upstream from RHEL, nowhere near as far upstream as Fedora, so a better candidate for dom0 IMHO. Qubes should stick with RH as that will have best chance for enterprise acceptance. Centos replacements like Rocky will probably come to the same end, as RH can make it arbitrarily hard (expensive) to follow.
So I fell down a rabbit hole a while back researching a related question: how large of a delay is there between security fixes being released in RHEL|mainline and uptake by Fedora or CentOS? Answer: itâs a trick question. Fixes originate from any number of places and circulate between Linux distros in a non-linear fashion before finally hitting your machine.
Since clones were already using CentOS as a synchronization point for patches that were (fingers crossed) close enough to RHEL they decided to just formalize the arraignment and switch CentOSâs nominal designation from being âdownstreamâ to being âupstreamâ.
Back to your assertion: we already are already about as far upstream from RHEL as you can get. So we wouldnât be losing stability. If you buy Oracleâs marketing bullshit of 100% binary compatibility (like I did) switching over requires fiddling with your DNF settings. But itâs a lie, they have to reverse engineer code drops from RHEL just like everyone else (old school CentOS included).
" Back to your assertion" - donât know which assertion you refer to.
Since Centos Stream is a rolling distribution, (rolling-minor, ha)
it cant be a serious candidate for dom0, regardless of its weight.
Yocto Project does not have a Security team ⌠there is some research and proof of concept work occurring with some tools but its struggling due to lack of people/resources.
And since this thread is about RPM distros ⌠what do you think about SilverBlue/CoreOS or SuSE MicroOS? SilverBlue and CoreOS leverage git to provide immutability, de-duplication, and rollbacks but leaves us the option to use DNF, which would make the transition easier. I know those arenât the only alternatives, but be wary of marketing fluff:
âŚ
This one:
Iâm skeptical that it will be any less stable than Fedora or what we can get for free from volunteers reverse engineering code drops from RHEL, like CentOS used to do. It also means security fixes take 24-72+ hours to hit the update servers. Seriously, go read the CentOS Blog about the new integrated release engineering process:
Whatever work that is under NDA or embargo, that will eventually get reverse engineered by Facebook (or whomever) and pushed onto CentOS Stream.
Do you have any input on letting users choose between CentOS Stream and a RHEL clone like Oracle or Rocky Linux?
Fedora Silverblue, CentOS Stream, or a RHEL clone all seem like acceptable options to me. Silverblue has the potential to reduce breakage, but I think that would have to wait until the GUI is out of dom0. The primary objection to RHEL was that some packages donât work and it is stuck on old software. But with a major release being cut every three years from Fedora via CI, it seems like a viable option.
I think that CentOS would be a good choice for dom0, as it can have a recent kernel, yet is extremely stable and has signed packages and signed metadata.
dom0 is in a rather weird position. It needs to be stable, so that changes donât catch the Qubes core team by surprise, but it needs the latest hardware support and installer. I doubt there are any existing distros that are good fits for us. A derivative of CentOS 8 is the best option I know of, but a fully custom distro would be ideal.
I agree. Iâm seeing that CentOS 8 Stream has moved forward a lot. I could redo some tests but well, that would deserves to be a specific task and @marmarek would need to arbitrate this.
Hm, that kind of misses the point. Let me try to be more verbose:
What does dom0 do currently?
hosts the GUI (XFCE)
hosts the audio subsystem (Pulse)
Kernel
XEN
other hardware is already isolated in sys-net and sys-usb
I think what makes most posters so passionate about which distro runs in
dom0 is their wish to have a different or more up-to-date GUI
environment. That goes away with the GUI VM.
Whatâs left? What does dom0 do in future?
Kernel
XEN
Qubes specific binaries
This is what I understand when I read âminimizing dom0â.
â There is no longer a need for a distro. â
Maybe I am wrong, but that is what I am expecting to happen and it looks
more secure to me than trusting any distro outside the Qubes project.
Experience shipping production systems matters. Alpine was shipping ânullâ as the default password for years.
A âdistroâ provides us with a critical mass of support, both in terms of eyeballs on the code and support from hardware vendors. Community distros havenât poured millions of dollars into testing and delivery infrastructure.
And aside from being shiny, what does Yocto or Alpine do that CoreOS doesnât?
And aside from being shiny, what does Yocto or Alpine do that CoreOS doesnât?
Again, I am not an expert. But the point as I understand it is the
reverse of your question. The goal would be a dom0 that is extremely
minimal, which means: reduced attach surface.
Currently we need a distro in dom0 because the GUI lives in dom0. Once
that is no longer the case, there should be no reason left for any user
to interact with dom0 in any way. There shouldnât be anything anybody
would want to install, configure or maintain in dom0. It should just be
the Kernel, XEN and Qubes daemons.
Thatâs what CoreOS is all about: reducing the base system to the maximum extent possible. Red Hat acquihired the leading container oriented distro and brought a bunch of process enhancements (like better release QA and back-porting) to make it more suitable for use in production.
The idea that Qubes doesnât need a base distro is silly: Linux is just a kernel and it doesnât make any internal stability guarantees. A distro is a coalescence of disparate upstream sources, which is a lot of work. Anyone can roll-their-own Linux distro badly, but its much easier to draft off of Debian or Red Hat.
Yocto and Alpine advertising themselves as minimal is less honest than the sticker price on a new car. 2.5 megabytes wonât get you running on Xen or boot a laptop. After you shove in everything required to actually do anything Yocto and Alpine are the same size as everyone else. The main difference is that they donât have nearly as many resources to make sure it all works correctly.
Even if dom0 doesnât require a GUI, we need a GUI somewhere and itâs more efficient if we use the same base system in dom0 as we do in the virtual machines. At a minimum, you get space savings by de-duplicating whatâs on disk. But compatibility is the biggest issue: we literally canât use CentOS right now because the userland is not in sync with Fedora and no one else has bothered to backport everything. Which leaves us stuck with a lot of extra porting work.
Ideally, we will someday get to use SeL4 and run everything in WASI isolates. But Qubes is not a research project and it needs to run on real hardware.
itâs more efficient if we use the same base system in dom0 as we do
in the virtual machines.
How and why?
At a minimum, you get space savings by de-duplicating whatâs on
disk.
Shared files between dom0 and a domU? I donât think so.
But compatibility is the biggest issue: we literally canât use
CentOS right now because the userland is not in sync with Fedora and
no one else has bothered to backport everything. Which leaves us
stuck with a lot of extra porting work.
I see. You want to use CentOS as domU and are frustrated by the amount
of work that needs to be done to get there. I can empathize with that.
That doesnât make it a good or better choice than what we already have.
In addition having a rolling distro by definition would be the opposite
of what we need in dom0. There should only be very very tightly managed,
infrequent and necessary (security & bug fix) updates to dom0 after release.
I lived on a rolling distro before (SuSe Tumbleweed). It was fun but not
stable in any sense of the word.
For the same reason that itâs more efficient to target a single execution environment in any other setting, be it browsers or specific versions of specific operating systems.
Like the Linux kernel or userland. You know, whatever is in the 130 megabytes of the Alpine Xen image ; )
No, I am not. I am talking about using the exact same version of hundreds of different pieces of software that makeup a distro. For example, there are at least 75 packages (see: [1, 2]) that Qubes uses in Fedora which are not available by default in RHEL.
What we have is Fedora, which is only stable for 1 year and porting between major releases is a resource drain on the project.
I will repeat this point one last time: you donât have to use CentOS Stream. The project could use Rocky Linux or any other RHEL clone.
CentOS Stream is downstream of Fedora and represents a minor point release for RHEL. So no major breaking changes, just new features and bugfixes. Iâve had Fedora stop booting on some laptops because it will switch between kernels without a major version bump. So I would imagine that CentOS Stream would be more stable than Fedora.
@Sven Iâm not finding any of your responses productive. You advocated for a minimalist distro, then switched to expressing dislike for CentOS Stream. I already addressed most of the concerns you raised, either in this thread or in the Github tickets. And you donât discuss alternatives that are based on an RPM or RHEL adjacent distro.
I get the impression you do not like Red Hat, which is fine. But this thread is focusing on RPM based distros. Not because I am a Red Hat fanboy, but because it means we donât have to replace all of our infrastructure at once. Some non-RHEL adjacent distro may be better for dom0. Thatâs fine, but please discuss that distro in another thread.
That being said, Iâm not wasting any more time entertaining low effort comments. I will flag anymore off-topic comments as such.
Itâs not âoff-topicâ, âlow effortâ, or ânot constructiveâ to question
the assumptions underlying this thread or your assertion that:
CentOS Stream surfaced as a top choice for a few reasons
In fact, it seems that following those discussions is far more
productive than a narrow focus on âwhat RPM distroâ, and productive in
the long run.
Identifying what needs to be in dom0, researching possible candidates,
and the work that would need to be done, thatâs useful. This isnât.
@Sven Iâm not finding any of your responses productive.
I am sorry to hear that.
It was my intention to share my understanding, which is that the choice
of distro in dom0 has very little impact beyond hardware support now and
will matter even less in the future.
We have Fedora now for historical reasons. The project lead pointed out
that once dom0 is minimized, we might get rid of Fedora in dom0 and
switch to something minimal.
In this context I recognized that we perceived things very differently
and it might be interesting to have a conversation to uncover incorrect
assumptions.
I get the impression you do not like Red Hat, which is fine.
My preferences boil down to a specific look & feel and stability (no
updates except security patches). I couldnât care less about the distro
or the packaging format in domU.
In dom0 I want no changes or user facing features at all unless they
significantly improve security.
this thread is focusing on RPM based distro not because I am a
Red Hat fanboy but because it means we donât have to replace all of
our infrastructure at once. Some non-RHEL adjacent distro may be
better for dom0. Thatâs fine, but please discuss that distro in
another thread.
I understood that from the start and tried to reason with you why
arguing for a distro change in dom0 might not be the best use of your
attention.
I largely started this thread and the Alt RPM Distro in dom0 ticket as a brain dump after doing a bunch of research. I was willing to rehash discussion from Github because I thought it would be helpful to have the discussion all in one place. But now it feels as if we are retreading ground already covered in this thread.
The vast majority of stuff in a Linux kernel is for hardware support. What is the testing infrastructure like for most distros? I would be surprised if anyone outside of Debian and CentOS have a good hardware testing infrastructure.
Define minimal: you can play at the margins, like replacing some GNU userland stuff with Busybox or a smaller standard C library. But those alternatives compromise on features and arenât going to be as well maintained. Since the application VMs will need the standard GNU userland and C library, we might as well just use the same distro in dom0 and dedupe the base image. At that point, it feels a lot like arguing about sports teams: itâs all the same code, just with different levels of integration and infrastructure.
Red Hat has expanded their free tier to 16 RHEL licenses for use in production; users unhappy with CentOS Stream could switch to RHEL for both dom0 and 15 appVMs without paying anything.
Red Hat has a free for FOSS infrastructure program; Qubes developers shouldnât need to worry about bumping up against the 16 cap of the aforementioned program.
I also realized that dom0 gets updated between ~1 and ~3.5 years (~1.9 years on average). If Red Hat sticks to their revamped 3 year release cycle, we wouldnât be stuck on old kernels much longer than what we are used to. The CentOS Hyperscale SIG provides accelerated backports (slides) of specific items (like systemd) and is working on backporting LTS kernels as well (which are released ~yearly). So it should be feasible to keep dom0 on a fresher kernel in future.
I think this could be something interesting to post on the qubes-devel mailing list so it more easily reaches the developers (only a few see the forum).
.
. Iâve also started up this