Release signed with wrong key (Publishing Infrastructure comprimise or documentation error)

Feedback
1

Currently, downloads of Qubes-R4.2.4 cannot be verified with the release keys, as described in the documentation:

Expected Keys

The documentation above clearly shows that the fingerprint of the release signing key is:

427F 11FD 0FAA 4B08 0123  F01C DDFA 1A3E 3687 9494
427f 11fd 0faa 4b08 0123  f01c ddfa 1a3e 3687 9494
427F11FD0FAA4B080123F01CDDFA1A3E36879494
427f11fd0faa4b080123f01cddfa1a3e36879494

Steps to Reproduce

I confirm that I already have this key on a new Fedora 34 DispVM:

[user@disp308 ~]$ gpg --import /usr/share/qubes/qubes-master-key.asc 
gpg: directory '/home/user/.gnupg' created
gpg: keybox '/home/user/.gnupg/pubring.kbx' created
gpg: /home/user/.gnupg/trustdb.gpg: trustdb created
gpg: key DDFA1A3E36879494: public key "Qubes Master Signing Key" imported
gpg: Total number processed: 1
gpg:               imported: 1
[user@disp308 ~]$ 

[user@disp308 ~]$ gpg --list-keys
/home/user/.gnupg/pubring.kbx
-----------------------------
pub   rsa4096 2010-04-01 [SC]
      427F11FD0FAA4B080123F01CDDFA1A3E36879494
uid           [ unknown] Qubes Master Signing Key

[user@disp308 ~]$

Then we download the Qubes-4.2.4 DIGESTS file, which has an inline GPG signature:

[user@disp308 ~]$ wget https://mirrors.edge.kernel.org/qubes/iso/Qubes-R4.2.4-x86_64.iso.DIGESTS
--2025-09-10 14:15:19--  https://mirrors.edge.kernel.org/qubes/iso/Qubes-R4.2.4-x86_64.iso.DIGESTS
Resolving mirrors.edge.kernel.org (mirrors.edge.kernel.org)... 147.75.80.249, 2604:1380:4601:e00::3
Connecting to mirrors.edge.kernel.org (mirrors.edge.kernel.org)|147.75.80.249|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1251 (1.2K) [text/plain]
Saving to: ‘Qubes-R4.2.4-x86_64.iso.DIGESTS’

Qubes-R4.2.4-x86_64 100%[===================>]   1.22K  --.-KB/s    in 0s      

2025-09-10 14:15:21 (30.4 MB/s) - ‘Qubes-R4.2.4-x86_64.iso.DIGESTS’ saved [1251/1251]

[user@disp308 ~]$ 

[user@disp308 ~]$ gpg --verify Qubes-R4.2.4-x86_64.iso.DIGESTS 
gpg: Signature made Mon Feb 17 00:00:00 2025 -05
gpg:                using RSA key 9C884DF3F81064A569A4A9FAE022E58F8E34D89F
gpg: Can't check signature: No public key
[user@disp308 ~]$

Here’s the full DIGESTS file

[user@disp308 ~]$ cat Qubes-R4.2.4-x86_64.iso.DIGESTS 
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256


6d28eed5e3a2f1e06b1dbdb52713deb8 *Qubes-R4.2.4-x86_64.iso
ede51181709b2e0a69b87cfd7b8db1f36ef69515 *Qubes-R4.2.4-x86_64.iso
f596adc4c40f2670321de0c41e65a4b94193ca77959149c62bcc2579425fac8e *Qubes-R4.2.4-x86_64.iso
949589751e8d4794274b4889dd94f50823b39011e2b8b0afa3b4b7fd204042baf640f1af4241dae42193cf33a9721600402a2f258397a2c81f20241576640607 *Qubes-R4.2.4-x86_64.iso
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEnIhN8/gQZKVppKn64CLlj4402J8FAmeywlAACgkQ4CLlj440
2J8jAg/+LY+/3fOyfbwjc8zkB+scnHv/BBxMB4o4hZrwFVfOt8d+ar34P1gvRDcG
GFkp7J4jjPU3Lrcdm2nzX0/uyrHrZq7TOvrWpyFn5IffyX8r2ltIF0SBld4Uvbhr
K8RPzJXTuFf1RrjcCngA24Go8aP0C4LL04PQNzCoXxMicqc+rDsk2zMQA/DdYE7S
2WZVkeuaUnRa31oxNJW7Di1UpPJxFBP/aTrii4e06hzqodtko+kxt9aAi8b47DO2
6g89H617zIXvcDfS/IAO8W7D+0If5WtLQyVGbeoNN+NOf14tE2wh18QtvVQmccVP
t23+aQz4AT6j49jDd0iUYbbcgyia2wvY4C4QfR5+ZaZaee5DfuJEgRIRBv+BQrCE
iG4o1HbsRXDpwZCHX3w99cVPpcYQzB0Xu1qR7zTNrXZBcgc5XffDoS0vzn57Id/7
wwXsSVTbdGjnFFffL8s5aOr5tH22QbqBcMYmF5OA/9QPdkX40oZDE6SFwRqQHS3t
OVsVAcv1r45NWWSRAjksviz91i7kMtl2KnP98imrdAUr/k21XupvxTVI5RgPF0DG
EWNIGsHZnSY9k2HqEl6+FYTH4hZBreaFGyASuk/lDjIYtGAz0IJERw6kFsIJ14YT
BGC8Kc0ysaLOcUCq4WTb0M1U8ePH41mtAxiz2LrLqT5MCkZ/yQI=
=YecE
-----END PGP SIGNATURE-----
[user@disp308 ~]$

What’s going-on here? Is the documentation wrong or is the publishing infrastructure compromised?

2

This is a fingerprint of a Qubes Master Signing Key (QMSK), not a release signing key (RSK):

Every Qubes OS release is signed by a release signing key (RSK), which is, in turn, signed by the Qubes Master Signing Key (QMSK).

2 Likes
3

Also for more info:

4

geez, this is a complicated process.

So, again, we first import the master signing key, which is pre-installed in the fedora 34 VM

I used the ubuntu gpg keyserver to fetch the key that the DIGEST was signed with

[user@disp2372 ~]$ gpg --import /usr/share/qubes/qubes-master-key.asc
gpg: directory '/home/user/.gnupg' created
gpg: keybox '/home/user/.gnupg/pubring.kbx' created
gpg: /home/user/.gnupg/trustdb.gpg: trustdb created
gpg: key DDFA1A3E36879494: public key "Qubes Master Signing Key" imported
gpg: Total number processed: 1
gpg:               imported: 1
[user@disp2372 ~]$

Then we download the latest release’s DIGEST file

[user@disp2372 ~]$ wget https://mirrors.edge.kernel.org/qubes/iso/Qubes-R4.2.4-x86_64.iso.DIGESTS
--2025-09-10 14:38:22--  https://mirrors.edge.kernel.org/qubes/iso/Qubes-R4.2.4-x86_64.iso.DIGESTS
Resolving mirrors.edge.kernel.org (mirrors.edge.kernel.org)... 147.75.80.249, 2604:1380:4601:e00::3
Connecting to mirrors.edge.kernel.org (mirrors.edge.kernel.org)|147.75.80.249|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1251 (1.2K) [text/plain]
Saving to: ‘Qubes-R4.2.4-x86_64.iso.DIGESTS’

Qubes-R4.2.4-x86_64 100%[===================>]   1.22K  --.-KB/s    in 0s      

2025-09-10 14:38:24 (22.4 MB/s) - ‘Qubes-R4.2.4-x86_64.iso.DIGESTS’ saved [1251/1251]

[user@disp2372 ~]$

Then we try to verify it, but we find it’s signed by some key whoose fingerprint is not referenced anywhere in the documentation.

[user@disp2372 ~]$ gpg --verify Qubes-R4.2.4-x86_64.iso.DIGESTS 
gpg: Signature made Mon Feb 17 00:00:00 2025 -05
gpg:                using RSA key 9C884DF3F81064A569A4A9FAE022E58F8E34D89F
gpg: Can't check signature: No public key
[user@disp2372 ~]$

So we fetch this key (DANGER!) from the ubuntu’s key servers

[user@disp2372 ~]$ gpg2 --keyserver-options no-self-sigs-only,no-import-clean --keyserver hkp://keyserver.ubuntu.com --recv-keys 0x9C884DF3F81064A569A4A9FAE022E58F8E34D89F
gpg: key E022E58F8E34D89F: 9 signatures not checked due to missing keys
gpg: key E022E58F8E34D89F: public key "Qubes OS Release 4.2 Signing Key" imported
gpg: no ultimately trusted keys found
gpg: Total number processed: 1
gpg:               imported: 1
[user@disp2372 ~]$

Now we have a potentially malicious key in our keyring :frowning:

But we use --list-sigs to see all of the signatures on this new, potentially dangerous key

[user@disp2372 ~]$ gpg --list-sigs
/home/user/.gnupg/pubring.kbx
-----------------------------
pub   rsa4096 2010-04-01 [SC]
      427F11FD0FAA4B080123F01CDDFA1A3E36879494
uid           [ unknown] Qubes Master Signing Key
sig 3        DDFA1A3E36879494 2021-11-29  Qubes Master Signing Key

pub   rsa4096 2022-10-04 [SC]
      9C884DF3F81064A569A4A9FAE022E58F8E34D89F
uid           [ unknown] Qubes OS Release 4.2 Signing Key
sig 3        E022E58F8E34D89F 2022-10-04  Qubes OS Release 4.2 Signing Key
sig          DDFA1A3E36879494 2023-06-03  Qubes Master Signing Key
sig          AC85E2284C05AA3E 2024-01-01  [User ID not found]
sig          AC85E2284C05AA3E 2024-01-03  [User ID not found]
sig          AC85E2284C05AA3E 2024-01-03  [User ID not found]
sig          0D9ED23637A736D5 2024-12-23  [User ID not found]
sig          0D9ED23637A736D5 2024-12-23  [User ID not found]
sig          B425658C7D1B5EBC 2025-01-31  [User ID not found]
sig          B425658C7D1B5EBC 2025-01-31  [User ID not found]
sig          B425658C7D1B5EBC 2025-01-31  [User ID not found]
sig          9F35D917E75FA8BE 2025-02-02  [User ID not found]

[user@disp2372 ~]$

We don’t have most of the keys that signed it, but we do see two: it signed itself, and our trusted Qubes Master Signing Key signed it. We use --check-sigs to confirm

[user@disp2372 ~]$ gpg --check-sigs
/home/user/.gnupg/pubring.kbx
-----------------------------
pub   rsa4096 2010-04-01 [SC]
      427F11FD0FAA4B080123F01CDDFA1A3E36879494
uid           [ unknown] Qubes Master Signing Key
sig!3        DDFA1A3E36879494 2021-11-29  Qubes Master Signing Key

pub   rsa4096 2022-10-04 [SC]
      9C884DF3F81064A569A4A9FAE022E58F8E34D89F
uid           [ unknown] Qubes OS Release 4.2 Signing Key
sig!3        E022E58F8E34D89F 2022-10-04  Qubes OS Release 4.2 Signing Key
sig!         DDFA1A3E36879494 2023-06-03  Qubes Master Signing Key

gpg: 3 good signatures
gpg: 9 signatures not checked due to missing keys
[user@disp2372 ~]$

And we check the Internet to figure out that the exclimation mark (!) indicates that it’s a good signature from one of the other keys in our keyring. Which, might not mean anything if you’ve previously imported untrustworthy keys.

I guess that works. But, geez, is there a better way to verify the authenticity of these RSKs before importing them into our keyrings?

5

You can go the following way to verify the Release Signing Key(RSK):

  • import and trust the Qubes Master Signing Key (QMSK)

  • check the Release Signing Key (without import) first:
    “gpg2 --show-keys --with-sig-list qubes-release-4.2-signing-key.asc”
    They you should get somewhat like

    grafik
    grafik812×122 6.67 KB

  • Then, you can check, that it is signed with the right key:
    “gpg -k DDFA1A3E36879494”
    grafik

  • If both checks successful, you can import the RSK
    “gpg --import qubes-release-4.2-signing-key.asc”

  • “gpg -k”

    grafik
    grafik706×173 7.15 KB

1 Like
6

Or you could do as the documentation suggests and get it from:

  • your existing Qubes installation (in /etc/pki/rpm-gpg/),
  • the downloads page (right next to the ISO, digests, and detached PGP signature files),
  • the qubes-secpack, or
  • the Qubes keyserver.

No need to check the Internet to figure this out. This is already explained on the doc page you linked.

It doesn’t matter whether you’ve previously imported any untrustworthy keys, since no other key can have the same fingerprint as the real QMSK, and that’s the only thing you have to trust. You don’t have to trust any of the other keys in your keyring.

On the other hand, maybe you’re referring to the possibility that importing a maliciously-crafted key could theoretically exploit a hypothetical bug in GnuPG (or whichever PGP program you’re using) or perhaps even compromise the entire system on which you’re carrying out this verification process. In that case, you have a few options:

  • You could follow @murdock’s steps above for checking the sigs on the RSK before importing it, but this still requires using gpg to significantly parse the untrusted key file, which is probably just as risky as actually importing it.
  • If you already have a Qubes installation, you could do these steps in a disposable qube with a fresh keyring.
  • You could get a USB drive with a physical write-protect switch and firmware that is either unflashable or cryptographically-signed (or both). Then, you could write the Qubes ISO onto this USB drive and flip the write-protect switch. It should now be very difficult for anything to alter what’s on the USB drive, even if you plug it into a compromised computer. You can now go around to a bunch of different computers (yours, your friends’, the library’s, an internet cafe’s, a computer store’s, etc.) and repeatedly re-verify the ISO that’s written on your write-protected flash drive.
1 Like