Ideas for next-generation qubes-builder

I’m far from being able to claim qubes-builder mastery (I’m much more at ease with the equivalent in eg. Debian and Yocto) – and I suspect the number of devs able to claim that is in fact quite low. The largest problem is probably the amount of custom code, which is not fully documented. Being essentially Makefile snippets probably does not help either.

As custom code, it requires attention from core devs for maintenance and evolution, which is not a good thing in a project with not many devs already. One item in the Design next major version of qubes-builder ticket left me thinking core devs could be on the same line on this.

Looking for a tool up to the task, that would provide features I’m missing from the current builder (eg. build-dependency handling, easy building of the same package for several distros) left me thinking, and a question asked to the right people brought me to Isar, which seems to provide many of the features we would need, except for a noticeable one: it currently targets Debian only - but then, there does not seem to be an structural problem preventing implementation of Fedora support. I found this ELCE '18 presentation to provide a nice summary, including comparison with similar projects (and a glimpse into their future through their collaboration in the EID project).

The basic idea is to rely on the flexibility of the bitbake build tool for build coordination. Bitbake is mostly known for its use in the Yocto embedded-distro framework, but is really a powerful standalone tool in itself, written in python, and allowing the use of python or shell code as is better suited. Isar (and ELBE, which has slightly less of the desirable features, as shown in the above slides) allow for exactly the kind of distro customisation we’re needing in Qubes. Note that the OpenEmbedded/Yocto recipes also show how various package formats (rpm, deb, ipk) can be produced from a single recipe; this could pave the way for people to get their preferred distro in dom0 and everywhere they would like.

I guess we could start to play with construction of Debian VM templates with this tool to get a feeling, and see from there if it seems worth to go on with Fedora support.

How does that sound ?

Update: there is also at least some interest in the Fedora world about those technologies, so the Qubes project would probably not end up maintaining just another single-usage builder.


Here is a quick overview of the bricks we would need to use ISAR, with the existing ones in blue. I did not include the generation of dom0 packages (not much different from template rpm’s, except for the chroot to use) and installer. PNG included for reader’s convenience, it seems we don’t have the graphviz discourse plugin available.

Building a rootfs is the core task if ISAR, and it is not too hard to build one following the instructions (for now I’ve used this for an uncustomized image, producing both a rootfs partition usable for a PVH and a legacy-bios boot image for HVM):

user@isar-build:~/isar$ . isar-init-build-env ../build
user@isar-build:~/build$ cat >> conf/local.conf 
MACHINE = "qemuamd64"
DISTRO = "debian-buster"
DISTRO_ARCH = "amd64"
WKS_FILE = "directdisk-isar.wks"
user@isar-build:~/build$ bitbake mc:qemuamd64-buster:isar-image-base

Basically the next things to do would be to create a chroot generator using mock instead of debootstrap, and a rpm packager to make use of it.

From there we should be able to work on individual tools packaging (including splitting complex CPU-hungry ones like stubdom rootfs generation to make the dev workflow smoother), which would make an ISAR-based Debian template package in reach, and a Fedora template could follow shortly.

digraph {
DebTemplate → DebRootfs → QubesDebs → DebPackager → DebBuildchrootGenerator;
DebRootfs → DebBuildchrootGenerator;
{ rank=same; QubesDebs → QubesDebPackaging; }
DebTemplate → RpmPackager;
RpmTemplate → RpmRootfs → QubesRpms → RpmPackager → RpmBuildchrootGenerator;
RpmRootfs → RpmBuildchrootGenerator;
{ rank=same; QubesRpms → QubesRpmPackaging; }
RpmTemplate → RpmPackager;

DebBuildchrootGenerator [color=blue];
DebPackager [color=blue];
QubesDebPackaging [color=blue];
QubesRpmPackaging [color=blue];