Building Qubes-ISO - Fedora 32 - dnf error python2-sh: Unable to find match

Hi,

I would like to build a Qubes-OS 4.0.3 ISO. I follow the guide:

except I do not do it on Qubes-OS, because currently I run it on a
Ryzen 3 dual core. So, I installed Fedora 32 on another laptop. So far
so good, but when I run the following command (taken from the guide):

sudo dnf install perl-Digest-SHA rpmdevtools rpm-build dialog rpm-sign
python2-sh dpkg-dev debootstrap devscripts

I get the error:

Error: Unable to find a match: python2-sh

Is as simple as that the Qubes-Builder is not compatible with Fedora 32
and I need to install Fedora 30?

For a hassle free build use Fedora 31.
There are requirements in the code which cannot be satisfied in Fedora
32. There’s an open issue referring to this.

For a hassle free build use Fedora 31.
There are requirements in the code which cannot be satisfied in
Fedora
32. There’s an open issue referring to this.

Ah…Thanks.

Installed Fedora 31 template; updated it; run again the dnf command and got the same error:

[user@fedora-31-R ~]$ sudo dnf install perl-Digest-SHA rpmdevtools rpm-build dialog rpm-sign python2-sh dpkg-dev debootstrap devscripts

Last metadata expiration check: 1:26:47 ago on Mon Sep 21 17:03:55 2020.
No match for argument: python2-sh
Error: Unable to find a match: python2-sh

I guess it has to be Fedora 30?

Have a look here please: https://github.com/QubesOS/qubes-doc/pull/1041

Have a look here please:
Update qubes-iso-building by fepitre · Pull Request #1041 · QubesOS/qubes-doc · GitHub

I will…Thanks.

Don’t the build scripts still require PyYAML? They did last time I
checked.

All should be migrated from python2 so it’s python3-pyyaml now. If you find the blocking point, please tell me where I will solve it.

Can you build on Fedora 32?

Yes all my dev VMs are Fedora 32 or some in 31.

Can you build on Fedora 32?

Qubes just finished to install the template. Now I’m updating it…I
long for my new machines…

@fepitre wrote:

Have a look here please:
Update qubes-iso-building by fepitre · Pull Request #1041 · QubesOS/qubes-doc · GitHub

I will…Thanks.

I did. There are 3 packages missing:

  • debootstrap
  • devscripts
  • dpkg-dev

That’s what make install-deps installs when executed in the AppVM.

According to the doc one shall install the dependencies in the AppVM,
but that means that that has to be done every time when one wants to do
a build. I did all the software installations in the template. Is that
a problem?

I’ve installed the Fedora-32 template and installed first the software
you proposed above: git createrepo rpm-build rpm-sign make python3-sh
rpmdevtools rpm-sign dialog perl-open python3-pyyaml perl-Digest-MD5
perl-Digest-SHA, and then created the AppVM. Ran make install-deps,
which installed the three packages (I mentioned above) and their
dependencies. So, I started the Fedora-32 template and installed
debootstrap, devscripts, and dpkg-dev. I shutdown the template, and
restarted my AppVM. Ran make install-deps again, and it is now happy.

I think in the doc it should state that all the dependencies should be
installed in the template and not in the AppVM. I think nobody wants to
have to install a whole bunch of software every time one wants to do a
build. Or do I miss here something?

The command make get-sources fails with:

→ Updating sources for dist-upgrade…
→ Fetching from GitHub - QubesOS/qubes-dist-upgrade release4.0…
→ Verifying tags…
No valid signed tag found!
fatal: No names found, cannot describe anything.
make: *** [Makefile:204: dist-upgrade.get-sources] Error 1

It should have been solved. A missing tag simply :slight_smile:

It’s why it’s primary said to use a standalone VM for building Qubes but for example, I do have installed on the needed dependencies into my template :slight_smile:

Nope…I just tried. I still get the same error.

Ok just notified Marek about it. He has certainly forgot to push the tag. You can simply remove the dist-upgrade component from the list. It’s useless for building Qubes 4.0.

Hi following a complete list of how I tried to build a the Qubes-OS 4.0.3 ISO. I list everything because I followed the documentation by the letter and it still fails at different points without a pattern or so. I tried in an AppVM where I installed all the dependencies in the template; failure. I tried on bare mental; failure. I tried in a standalone VM; failure. I tried also with pre-build packages and without; failure.
So, by now I doubt that the procedure, which is outlined in the documentation works for someone who starts from scratch.

Today I deleted everything and started from scratch again on Qubes-OS 4.0.3:

  1. Installed Fedora 32 template.
  2. Updated that template via dnf update.
  3. Created a standalone qube based on that template.
  4. Build the Qubes Builder environment:
    sudo dnf install git createrepo rpm-build rpm-sign make python3-sh rpmdevtools rpm-sign dialog perl-open python3-pyyaml perl-Digest-MD5 perl-Digest-SHA
  5. Created a working directory.
  6. Followed the document “Building Qubes OS ISO” by the letter. I used the setup.sh script. I chose release 4, and I chose pre-build pkg. Then I executed:
    make install-deps
    make get-sources

So far everything was okay. I added to the builder.conf COMPONENTS := $(filter-out gcc,$(COMPONENTS))

Then I executed:
make qubes

Which failed with this:

make[1]: Entering directory ‘/home/user/My-Stuff/GitHub/Qubes-OS/qubes-builder’
git -C /home/user/My-Stuff/GitHub/Qubes-OS/qubes-builder/chroot-dom0-fc25/home/user/qubes-src/mgmt-salt clean -f -d -X
/home/user/My-Stuff/GitHub/Qubes-OS/qubes-builder/scripts/create-archive /home/user/My-Stuff/GitHub/Qubes-OS/qubes-builder/chroot-dom0-fc25/home/user/qubes-src/mgmt-salt qubes-mgmt-salt-4.0.22.tar.gz
~/My-Stuff/GitHub/Qubes-OS/qubes-builder/chroot-dom0-fc25/home/user/qubes-src/mgmt-salt ~/My-Stuff/GitHub/Qubes-OS/qubes-builder
~/My-Stuff/GitHub/Qubes-OS/qubes-builder
Traceback (most recent call last):
File “/home/user/My-Stuff/GitHub/Qubes-OS/qubes-builder/qubes-src/mgmt-salt/yaml-dumper”, line 18, in
import yaml
ImportError: No module named yaml
make[1]: *** [qubes-src/mgmt-salt/Makefile.builder:59: mgmt-salt-create-vars-makefile] Error 1
make[1]: Leaving directory ‘/home/user/My-Stuff/GitHub/Qubes-OS/qubes-builder’
make: *** [Makefile:239: mgmt-salt-dom0] Error 1

I checked /home/user/My-Stuff/GitHub/Qubes-OS/qubes-builder/qubes-src/mgmt-salt/builder.conf:

ifeq ($(PKG_MANAGER),dpkg)
DEPENDENCIES += python3-yaml
else ifeq ($(PKG_MANAGER),rpm)
DEPENDENCIES += python3-pyyaml
endif

I checked /home/user/My-Stuff/GitHub/Qubes-OS/qubes-builder/qubes-src/mgmt-salt/yaml-dumper:

#!/usr/bin/python2 -O

:maintainer: XXXXXXXXXXXXXXXX
:maturity: new
:depends: none
:platform: all

Dump a YAML configuration file to key = value pairs
‘’’

import argparse
import collections
import os
import sys
import yaml

There seems to be a mismatch. mgmt-salt builder.conf sets the dependency to python3, but yaml-dumper is executed with python2.

To me the Qubes-Builder behaves like rolling the dice. What branch should I use? The setup script does not offer a specific branch like 4.0.3. Which branch is locked (no changes are applied)?

There is an open issue on PyYAML treatment in Fed32 builder, but other
users report successfully building.
Qubes-Builder should (and did) work perfectly, which ever branch
you use. 4.1 is still a development branch, so you might encounter
errors there, but 4.0 should be rock solid. (If you build using 4.0 you
will end up with 4.0.3 with all available upgrades.)
NO branch is locked, since even 4.0 is still receiving updates, as bugs
are squashed.

There is an open issue on PyYAML treatment in Fed32 builder, but
other

Ah…Haven’t seen it. Could you provide the issue number?

users report successfully building.
Qubes-Builder should (and did) work perfectly, which ever branch
you use. 4.1 is still a development branch, so you might encounter
errors there, but 4.0 should be rock solid. (If you build using 4.0
you
will end up with 4.0.3 with all available upgrades.)
NO branch is locked, since even 4.0 is still receiving updates, as
bugs
are squashed.

Okay, I thought so, but since I experience such erratic behavior I
started wonder; even doubting myself. However I provided the steps I
did. Could you confirm that I didn’t miss something? Is that the
correct process to build the ISO release 4?