Building Qubes - trying to get a first successful build

I’m trying to get Qubes to build using the Qubes executor. As I haven’t managed to get any successful build yet, I’m trying to follow the README strictly, and building on fresh vms. I’m using examples/qubes-os-r4.2.yml unchanged from the current main (6942fa).

Somehow I already failed, since my build qube has a hostname of fedora rather than the qube name, which causes qubes/executors/ to break (but hacking that works). (Later, found the cause: clone a qube from fedora-39-xfce and include an underscore in the name, and its hostname will be fedora. That’s unexpected. Fixing this didn’t change the following.)

Now I have qb package fetch failing trying to verify python-fido2, because the signing key 20EE325B86A81BCBD3E56798F04367096FBA95E8 is bad (expired in 2020). I can also see this in the build logs for fedora and debian last month, but I’ve no other context. It recurs if I remove artifacts/sources and re-run.

Same error with qubes-os-main.yml

Chasing down the individual error would be good, but I’m mainly posting because I’m confused about why I’m having problems. It looks like the automated CI runs several times a day, and I’m guessing there are other users here running builds daily, plus the core team. The docs look clear to me and there doesn’t seem to be much room to mess things up. I also hit several other obstacles a couple of weeks ago running the Docker executor (specifics not recorded, but not related to key signing). I haven’t seen much on this forum about builds breaking.

So I think I’m looking for more context. Is running off main recommended, or is there some baseline that might be more reliable? Are there big differences between a local build and the automated CI? Are both executors stable and supported? Is there another place where build issues are discussed?

Same error with a fresh build a week later. I’ve verified it also fails the same way with the docker executor.

This is a problem for me participating on the audit project. There, I’m interested in lowering unnecessary barriers to entry to get more people hands-on. Running into difficulties with the default build, and not getting responses here, conflicts with that.

List of steps

  1. In a new template cloned from stock fedora-39-xfce, clone qubes-builderv2 under /opt
  2. Copy qubes-os-r4.2.yml to builder.yml
  3. Install packages from dependencies-fedora.txt
  4. Install mkmetalink using the qubes-infrastructure-mirrors readme and move to /usr/bin
  5. Do executor-specific stuff
  6. In a new qube based on the new template, run ./tools/ docker fedora-39-x86_64 and add the default user to the docker group
  7. Run ./qb package fetch

I have the same issue on Fedora 40 (outside Qubes OS). I removed python-fido2 from the dependency list, but then I encountered the same issue with lvm2. Removing both of them makes it work but I guess lvm2 is mandatory.
Fetch and Prep stages work but I then have an error on Build, so same, not able to build an ISO for now.

I’m using Podman because it seems to work better than Docker on Fedora.

Also the doc gives the example with a Fedora 36 Docker image but it’s not working anymore with Fedora 40 host. Not specifying mock configuration allows to build latest.

I’m not surprised, last year I’ve been working with the builder and there were also similar issues with wrong config files. It wasn’t difficult to fix but I guess devs are using their own configs and we need more contributors.

1 Like

It’s weird because the log says:

sqv --keyring /tmp/tmp.Jzo3DW0BTO/keyring /home/user/Qubes/qubes-builderv2/artifacts/tmp/tmp159fgf5n/untrusted_fido2-1.1.2.tar.gz.sig /home/user/Qubes/qubes-builderv2/artifacts/tmp/tmp159fgf5n/untrusted_fido2-1.1.2.tar.gz
Signing key on 20EE325B86A81BCBD3E56798F04367096FBA95E8 is bad:
            The primary key is not live
   because: Expired on 2020-05-01T11:07:16Z

But if we look at the .qubesbuilder config file from qubes-python-fido2 (qubes-python-fido2/.qubesbuilder at main · QubesOS/qubes-python-fido2 · GitHub), download the release manually (Release python-fido2 1.1.2 · Yubico/python-fido2 · GitHub) and the corresponding PGP key ( to manually verify the file, then we have a good signature with the key expiring in 2025.

1 Like

So of course, the problem comes from the fact that the public keys used to verify are here: qubes-python-fido2/debian-pkg/debian/upstream/signing-key.asc at main · QubesOS/qubes-python-fido2 · GitHub and the one that fails (BA95E8) expires in 2020 in this file. It has been renewed I guess and we need to update it.

But then we are back at the original question, how someone can currently build Qubes OS :thinking:


Appreciate the extra data points here, thanks.

I guess devs are using their own configs and we need more contributors.

My reasons for assuming upfront that the build is more mature were:

  1. There’s a detailed developer section on the front page of the website documentation.
  2. There’s a testing community.
  3. There’s a daily CI build (I can’t view the GitLab page, but I can see the code and the logs. At a glance, the CI code appears to do more than simply wrap the builderv2 code.).

I didn’t look deeply, but couldn’t find a recent change to explain this. Also, why the CI build hit this failure once in April but not since.

I opened an issue: builder-v2: fetch stage fails due to incorrect public key for `python-fido2` · Issue #9289 · QubesOS/qubes-issues · GitHub

Here is how to fix it locally:

  • Download the new key (
  • mv ~/Downloads/20EE325B86A81BCBD3E56798F04367096FBA95E8.asc artifacts/sources/python-fido2/debian/upstream/signing-key.asc

But now we have another issue with LVM2:

sqv --keyring /tmp/tmp.6iZBXPWQav/keyring /home/user/Qubes/qubes-builderv2/artifacts/tmp/tmpp2l3ncip/untrusted_LVM2.2.03.09.tgz.asc /home/user/Qubes/qubes-builderv2/artifacts/tmp/tmpp2l3ncip/untrusted_LVM2.2.03.09.tgz
Verifying signature:
            Policy rejected non-revocation signature (Binary) requiring collision resistance
  because: SHA1 is not considered secure
1 Like

This time it’s because the key uses SHA1 and it’s flagged by sequoia (Return error messages · Issue #39 · rpm-software-management/rpm-sequoia · GitHub).
Marmarek did something similar 2 months ago (Release signing key still uses SHA1 · Issue #7124 · fwupd/fwupd · GitHub and Update signing key · QubesOS/qubes-fwupd@923abd6 · GitHub).

New key is here:

But replacing the key (mv ~/Downloads/D501A478440AE2FD130A1BE8B9112431E509039F.asc artifacts/sources/lvm2/mcsontos-key.asc) does not work, there is this issue:

Signing key on D501A478440AE2FD130A1BE8B9112431E509039F is not bound:
            No binding signature at time 2020-03-26T11:26:45Z

See Importing a key has no verbosity information, unhelpful error message · Issue #1974 · rpm-software-management/rpm · GitHub

1 Like

If you disable the sqv check in qubesbuilder/plugins/fetch/scripts/verify-file (add || true to the end), do you get a complete build, or more issues?

The problem comes from sqv not allowing SHA1 verification. Marmarek had this issue already multiple times (Use sqv for file verification, second attempt · QubesOS/qubes-builderv2@fae0db1 · GitHub, Add option to allow signing keys bound with SHA1 (#4) · Issues · sequoia-pgp / sequoia-sqv · GitLab). I think going back again to gpgv for temporary development is enough to move on and keep working. I opened an issue already.

What I described above make Fetch and Prep stages work, but then Build fails with this:

mknod: /builder/pbuilder/build/37/test-dev-null: Operation not permitted
 E: Cannot install into target '/builder/pbuilder/build/37' mounted with noexec or nodev
 E: debootstrap failed

Can someone else confirm?

How far into the build is this? I have a ./qb package fetch prep build 9 hours in currently after disabling sqv. (this is not the furthest I’ve got, unfortunately)

I also wonder if there’s anything that can be done about network resilience - I often get github network failures over whonix, and I think my ISP blocks, so I only get stability over VPN right now. Not ideal.

I get the error just after running the build script.

I had similar issues with network also, I agree, but there is also an option not to refetch sources: skip-git-fetch: true in the builder.yml, to save time on the next run.

1 Like

Most of my 9+ hours has been in prep, not got to build yet.

Oh yeah, it was more like 3-4 hours for me but I’m not running it on Qubes but on a Fedora 40 install so much faster. The 4.2 config file also contains way too many stuff for testing, I started to create one with the bare minimum

Ah that’s useful. I’ll consider paring down the component list after getting one successful build with everything. Did I miss some docs for building for testing somewhere?

I don’t see anything on the doc about testing configs, but for example it’s not necessary to build other DE than XFCE if you want to test a basic install. Separate packages by categories like “Base”, “KDE”, or “Full” for example will be useful later to create groups in Anaconda and have more choice during install

No immediate failure on build - looks like some components build successfully, before failing on rpm-oxide.

[executor:qubes:disp4136] output: DEBUG: error: Empty %files file /builddir/build/BUILD/qubes-rpm-oxide-0.2.7/debugsourcefiles.list

[executor:qubes:disp4136] output: ERROR: Exception(/builder/build/qubes-rpm-oxide-0.2.7-1.fc39.src.rpm) Config(fedora-39-x86_64) 12 minutes 26 seconds

Going to preserve the environment and maybe come back to this later.