Hey guys. Thanks for being interested in, just to remind you that this topic is How did you, and not How to.
That would be installing qubes 4.1 entry under Step 4 - Installing Qubes and other OSes - Heads - Wiki as referred previously.
This is standard thin LVM pool over LUKSv2 encrypted container, with fedora template used for qubes, Whonix used for updates (dom0, templates) and disposable sys-firewall, sys-usb and sys-net.
@enmus thanks for the friendly reminder.
Could you possibly share how did you install Qubes? From the USB stick? I am genuinely interested in this, how to do it most securely today, with modern computers…
Both previous references says that it is directly from iso and detached signature file sitting side by side on an ext3/ext4 partition… I feel people don’t read.
The qubes section also shows, with screenshots, the booting and iso verification process.
@enmus: what is missing there?
I guess you could already guess what…
Thanks again for repeating in order to be more clear to me. Very nice idea though, while I still have concerns about using USB sticks!
Release signing key (public key counterpart) is included in the firmware, as documented under the general section of the same page.
This is how Tails, Arch Linux, Qubes OS can boot directly from and ext3/ext4 formatted drive, if iso and iso.asc/iso.sig detached signature are sitting side by side on the USB thumb drive root directory.
If distros were detach signing their iso (Fedora stopped) those distros could also be booted the same way, at the condition of having their distro signing public key added under Heads and present in booted firmware to be able to verify corresponding detached signatures alongside of downloaded isos.
No magic there, the iso.asc/iso.sig detached signature is validated against the distribution public key that is provided under Heads. This is why Heads can validate both integrity and authenticity of those ISOs, just as their webpages describe to do so manually, but automatically under Heads. And since Heads validates those distribution signing public key as part of Heads firmware measured boot, the user can simply drop newer isos and detached signatures files and boot from them confidently.
Bonus, Qubes daily produced QA builds can also be booted the same way, since that corresponding public key is also under Heads since Maximized builds were merged under Heads.
Well, if you dd an image over USB thumb drive, you loose integrity validation. This is a worry-less alternative approach for Heads users.
Also, you could detach sign an ISO yourself from Heads if the distribution do not provide those
- verify integrity manually from Heads recovery shell (mount-usb rw, cd /media, sha256sum distrib.iso, verify hashes manually.)
- detach sign with your own private key (gpg --card-status, gpg --detach-sign distro.iso)
- Sync changes (cd /, umount /media)
- boot your own detached signed iso.
That way, you can trust that the iso is as expected prior of booting into it. Those “public” files can sit side by side alongside of other “public” (untrusted) files, where heads will verify them prior of mounting them and boot into them.
Qubes 4.1.1 release signing key, in Qubes download website,
are longer than Qubes distro key, in heads Github repo.
is it expected, or anything wrong ?
Will try to answer factually, which requires me to redo the process.
As you can see on heads/initrd/etc/distro/keys at master · osresearch/heads · GitHub , the latest time heads/qubes-4.key at master · osresearch/heads · GitHub was touched is linked to commit Separate trusted ISO signers from trusted config signers · osresearch/heads@c1be56c · GitHub which doesn’t give much information on what was applied to the public key to reduce its size 5 years ago.
So let’s attempt to replicate the work done there with tails.key which I had to update 10 months ago heads/tails.key at master · osresearch/heads · GitHub linked to commit tails.key : merging of new long-term signing key with old one so old … · osresearch/heads@7a324bb · GitHub which I left traces for reproducibility under the commit message, pointing to Update Tails public signing distro key (expires on 2022-01-19) by tlaurion · Pull Request #1023 · osresearch/heads · GitHub
So let’s replicate Update Tails public signing distro key (expires on 2022-01-19) by tlaurion · Pull Request #1023 · osresearch/heads · GitHub with qubes distribution public key (release key) and compare things on a dvm:
[user@disp2326 ~]$ cd /tmp [user@disp2326 tmp]$ mkdir qubes-key [user@disp2326 tmp]$ cd qubes-key/ [user@disp2326 qubes-key]$ wget https://keys.qubes-os.org/keys/qubes-release-4-signing-key.asc --2022-11-16 12:26:43-- https://keys.qubes-os.org/keys/qubes-release-4-signing-key.asc Resolving keys.qubes-os.org (keys.qubes-os.org)... 184.108.40.206, 2604:1380:4601:1c00::1 Connecting to keys.qubes-os.org (keys.qubes-os.org)|220.127.116.11|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 2379 (2.3K) [application/octet-stream] Saving to: ‘qubes-release-4-signing-key.asc’ qubes-release-4-sig 100%[===================>] 2.32K --.-KB/s in 0s 2022-11-16 12:26:44 (49.2 MB/s) - ‘qubes-release-4-signing-key.asc’ saved [2379/2379] [user@disp2326 qubes-key]$ gpg --home /tmp/qubes-key/ --import qubes-release-4-signing-key.asc gpg: WARNING: unsafe permissions on homedir '/tmp/qubes-key' gpg: keybox '/tmp/qubes-key/pubring.kbx' created gpg: key 1848792F9E2795E9: 1 signature not checked due to a missing key gpg: /tmp/qubes-key/trustdb.gpg: trustdb created gpg: key 1848792F9E2795E9: public key "Qubes OS Release 4 Signing Key" imported gpg: Total number processed: 1 gpg: imported: 1 gpg: no ultimately trusted keys found [user@disp2326 qubes-key]$ gpg --home /tmp/qubes-key/ --edit-key 1848792F9E2795E9 gpg: WARNING: unsafe permissions on homedir '/tmp/qubes-key' gpg (GnuPG) 2.3.7; Copyright (C) 2021 Free Software Foundation, Inc. This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. pub rsa4096/1848792F9E2795E9 created: 2017-03-06 expires: never usage: SC trust: unknown validity: unknown [ unknown] (1). Qubes OS Release 4 Signing Key gpg> minimize User ID "Qubes OS Release 4 Signing Key": 1 signature removed pub rsa4096/1848792F9E2795E9 created: 2017-03-06 expires: never usage: SC trust: unknown validity: unknown [ unknown] (1). Qubes OS Release 4 Signing Key gpg> quit Save changes? (y/N) y [user@disp2326 qubes-key]$ gpg --home /tmp/qubes-key/ --export --armor 1848792F9E2795E9 > qubes-release-4-signing-key_mod.asc gpg: WARNING: unsafe permissions on homedir '/tmp/qubes-key' [user@disp2326 qubes-key]$ wget https://raw.githubusercontent.com/osresearch/heads/master/initrd/etc/distro/keys/qubes-4.key --2022-11-16 12:31:22-- https://raw.githubusercontent.com/osresearch/heads/master/initrd/etc/distro/keys/qubes-4.key Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 18.104.22.168, 22.214.171.124, 126.96.36.199, ... Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|188.8.131.52|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 1629 (1.6K) [text/plain] Saving to: ‘qubes-4.key’ qubes-4.key 100%[===================>] 1.59K --.-KB/s in 0.001s 2022-11-16 12:31:22 (1.58 MB/s) - ‘qubes-4.key’ saved [1629/1629] [user@disp2326 qubes-key]$ ls -al total 72 drwxr-xr-x 3 user user 200 Nov 16 12:31 . drwxrwxrwt 13 root root 320 Nov 16 12:29 .. drwx------ 2 user user 40 Nov 16 12:27 private-keys-v1.d -rw-r--r-- 1 user user 1312 Nov 16 12:29 pubring.kbx -rw-r--r-- 1 user user 1890 Nov 16 12:27 pubring.kbx~ -rw-r--r-- 1 user user 1629 Nov 16 12:31 qubes-4.key -rw-r--r-- 1 user user 2379 Feb 8 2022 qubes-release-4-signing-key.asc -rw-r--r-- 1 user user 1611 Nov 16 12:30 qubes-release-4-signing-key_mod.asc -rw-r--r-- 1 user user 49152 Nov 16 12:29 tofu.db -rw------- 1 user user 1200 Nov 16 12:27 trustdb.gpg
So we see that the 3 keys are different.
Let’s see how/why, comparing heads downloaded key with the one we just generated:
[user@disp2326 qubes-key]$ diff qubes-4.key qubes-release-4-signing-key_mod.asc 2d1 < Version: GnuPG v1
Basically, the generated one is the same as the old one, without the GPG v1 header and is considered equal.
Then lest’s compare heads downloaded one with the one coming from Qubes OS website:
[user@disp2326 qubes-key]$ diff qubes-4.key qubes-release-4-signing-key.asc 2d1 < Version: GnuPG v1 27,28c26,39 < mqH0I+V+LuMtlE521YHKg0tsB9GVlfWBS10= < =QN1j --- > mqH0I+V+LuMtlE521YHKg0tsB9GVlfWBS12JAjMEEAEKAB0WIQRCfxH9D6pLCAEj > 8Bzd+ho+NoeUlAUCYaQmlwAKCRDd+ho+NoeUlB40D/0YwLGqX5O6tl/q0Vehud2N > mm5OIpxSZKrpm8vNtf2/rzumBldFSczCtVAkHo4N23hC+IGKHSG7lFZlFue/cng0 > ngopJsfhbj8eAbtdo9lqiqQaiFtUrB8hTd1HgvHjCptBKrSKn4FlJJ91ypLkoyiX > 27TcfToyEq6qFAWKXXQosYtCzh492WlD7GXXz32/1LnZKMS3TR4x+QfVRc9kn8X5 > HaempDgWw79d7ZAcSDuO4Kb2j/se4aLESTefKtJJ9LuPqhHZ+qGekUCyweiZ+mkR > ok6XcaOHJEJgsvG1DIGwrGXyKkvqi12W8Q95XAF7y7C98vq4cVyhiLrBCbdgMY4X > l9vHXIVEL3C032qu4AaeJ2tHZJvl1+nYqam8urQ/APk6pPVs75+IkH7zDfEHh+SF > m6xAg/0fMKhYDlXB7l3UbiioV8vhBlHEf4XFR/VnnqM1B1TwdjywAbenc9Ev45L0 > 5oqfrerACvYMxUxQVR0WQyrsLjJzinKZjjZW5KYiRn9lO+27rp1kMQHQFhNX1pnj > oVnJIi0AtwAXumKI78SQpVRMjj6+fqusI/1Zx63XcI9Y8BBZvssYscgt2c9oMtBT > B/Ng9EaplpSSVCsdcN85VAR7CZT5YPsvsMzSmmUUIRoEb/dHMuMmh0YbBJbBUikP > dcBETivefvOIZRjSyZYUTg== > =6A62 [user@disp2326 qubes-key]$
Or more visually through meld:
Basically, gpg “minimize” operation of the key removed an additional signature (the release key being signed externally) which can be seen:
[user@disp2326 qubes-key]$ gpg --home /tmp/qubes-key/ --import qubes-release-4-signing-key_mod.asc gpg: WARNING: unsafe permissions on homedir '/tmp/qubes-key' gpg: key 1848792F9E2795E9: "Qubes OS Release 4 Signing Key" not changed gpg: Total number processed: 1 gpg: unchanged: 1 [user@disp2326 qubes-key]$ gpg --home /tmp/qubes-key/ --import qubes-release-4-signing-key.asc gpg: WARNING: unsafe permissions on homedir '/tmp/qubes-key' gpg: key 1848792F9E2795E9: 1 signature not checked due to a missing key gpg: key 1848792F9E2795E9: "Qubes OS Release 4 Signing Key" 1 new signature gpg: Total number processed: 1 gpg: new signatures: 1 gpg: no ultimately trusted keys found
Under Heads, we keep only the public key used to verify the detached signature. Any other signee which signed the key is not important, since the validity is already considered “trusted” per Heads committer (which signed his commit) and we try to reduce the size of the public key that are injected in the firmware, removing bloating stuff.
This is not really important in the gained size for Qubes release public key, but as can be seen for tails.key, not having applied gpg’s
minimize on their ditrribution signing key (release public key) would have bloated us of 2191 additional signee, basically saying that recognisez the validity of the public key by apposing their signatures to it… We do not need that.
In short: Heads simply applied the same minimizing policy to all public key included in the firmware to limit bloating. Public keys are basically high entropy blobs, and are not compressed as part of Heads initrd packing otherwise. And since space is precious, anything that can be removed is removed
And if we wanted to replicate the process without the warnings of keys, we would have to follow Verifying signatures | Qubes OS to get rid of the warnings above and add trust to Qubes master key, which is actually the signee we removed from qubes-4.key by above process, and explains the difference in size of the file uploaded under Heads github and used to verify detached signed iso!
i see, okay, thanks for information @Insurgo
by the way, since you know a lot about heads, are you part of Heads development ?
i also see, apparently you add verifying installation into Heads documentation.
also, i tried to USB boot from external DVD,
but apparently Heads can only recognize USB thumb drive,
is it possible, for Heads, to USB boot from external DVD ?
if i understand correctly,
the release signing key, under Heads, will be used to verify detached PGP signature,
then the detached PGP signature, will be used to verify Qubes ISO. is it correct ?
but what i don’t understand is, whether it is done automatically by Heads ?
or we have to do it manually in the Heads recovery shell ?
in case the verification fails, will Heads give us kind of warning ?
i.e. release signing key under Heads, fail to verify detached PGP signature, in the media,
or detached PGP signature, fail to verify the ISO.
and then, in case the USB thumb drive we use is infected / compromised,
will it infect heads ?
thanks and regards,
@deeplow: please split the topic from here
It’s about Heads
Yes, i’m part of Heads development. I’m tlaurion · GitHub
Would it be possible to boot from external DVD? Yes, for some platforms with docking station and/or USB + and compiling + packaging those kernel drivers into initrd so that those could be loaded on demand. That would consume space in the firmware, while i’m not sure this is really needed? If you think so, please open an issue on Issues · osresearch/heads · GitHub, it was never asked before.
Yes, Heads can currently only “USB boot” from USB drives only. Supporting USB DVD would require SCSI support and other modules to be packed inside of Heads, where booting from signed ISO from a USB thumb drive fits the bill while not requiring to burn a DVD at each new ISO version, which seems to be a better low cost approach. A lot of “things” can be supported by USB, but Heads basically supports USB2/USB3 controllers by default (with USB1 support needing explicit board configuration) through generic drivers, loaded on demand. Then usb-storage when needed to detect block devices on top of USB. Then the kernel sees no difference, abstracting SATA/NVMe/USB drives. But all other USB devices kernel drivers are not compiled in the kernel nor compiled as modules; hence not making the hardware available to Heads. Also, there is no udev or systemd under Heads. The scripts are specifically loading the required drivers when needed, and prior of loading them, measuring them, so that if for some reason an additional driver was loaded, Heads TPM measurements would be different and would not unseal either TOTP/HOTP nor the TPM Disk encryption key to default boot the machine on normal boot.
A GPG detached signature (that .iso.asc/iso.sig in case of USB ISO boot) is basically GPG generating a hash that is then verified against a signed one, that signed one being under .iso.asc/iso.sig. It’s basically done in one pass from gpg --verify against a detached signature. In the case of iso, you can see per the size of the iso.asc/iso.sig that a lot of “blocks” are considered in the detached signature, which is proportional to the size of the content that was detached signed with the release private key on Qubes OS side. So verifying a file (or a digest: which is basically a list of checksums+files), needs to be filled in the same order to be validated through a detached signature. You might want to dig deeper into signatures and detached signatures differences if needed What is the difference between a GPG signature and a detached signature? - Unix & Linux Stack Exchange
Step 4 - Installing Qubes and other OSes - Heads - Wiki gives a complete example of output of what you would have got on screen when Qubes 4.1 iso detached signature verified both integrity and authenticity of the iso file, prior of stating its validity, and then if valid only, propose the boot options of what was found under the grub.cfg file under mounted iso. If the detached signature doesn’t match, ISO boot from USB will simply fail, not permitting to boot from a corrupted/non-authentified ISO file against its detached signature.
In the case you download a ISO that is not providing a detached signature, instructions are provided to generate one yourself, of course after having verified the integrity of the ISO yourself prior of signing it. This way, you can have multiple ISOs side by side, ones being supported by Heads will boot them after having their authenticity/integrity against public keys present in the firmware, and the ones you detached signed yourself against your public key also fused inside of the firmware.
Heads doesn’t pretend to have magic preventing vulnerability exploitation that would only live in memory based on exploitation of its packed tools. Heads verifies that the measured and sealed integrity of the firmware (TOTP/HOTP) is consistent across boots. That is, after an OS is installed. Rubber ducky types of attacks are mitigated on boards not requiring USB Keyboard (usb HID support) on boot, which could go in recovery and back through emulation of keyboard keypresses. Anything that would tamper the firmware (coreboot+Heads payload) will be detected on next boot if TOTP is sealed. TOTP/HOTP unsealing will not work if coreboot measured stages are not consistent to what was sealed when generating a Qr code/pairing HOTP with user’s USB Security dongle.
So, for example, if your USB thumb drive firmware is targeting you, and for whatever reason faking the reading of blocks and made gpg report the detached signature as being valid (that would be amazing), but then your USB thumb drive replaced the ISO file that was validated with a tampered one(also amazing), there is no way Heads could detect this and would do the kexec call to the mounted ISO, passing magic related to the path/GUID of the ISO/other magic depending on the linux distribution at play and tweaks needed to boot its installer (this varies across debian/fedora/archlinux etc).
When one decides to boot from USB, usb-storage kernel module is additionally loaded on demand. The attack surface of USB devices at that point is a combination of a USB thumb drive tampered firmware, the USB controller drivers inside of Heads, the Heads kernel’s filesystem drivers (ext2/ext3/ext4/Jolliet(ISO)/), USB thumb drive filesystem exploited bugs, and busybox from the versions packed under flashed Heads version used. On top of that, if detached signature are validated, we add the gpg toolstack tools packed under Heads.
What I want to put emphasis on here is the following: Heads is a ram based Recovery Shell, executed from a measured boot coreboot, where Heads also measures its modules prior of permitting to unseal an additional TPM Disk encryption key sealed into a separate TPM’s NVRAM memory region, up to the point of booting an OS (USB, Disk). After that, Heads linux kernel’s space is replaced (kexec) with loaded kernel+initrd+xen binaries, and Heads is replaced by the OS booted environment.
An “infected/compromised thumb drive” can mean many things, most of the time containing untrusted files, possibly infected per another OS the USB thumb drive was connected to. Booting from ISO+detached signature is a simple but additional and effective way to make sure that the ISO in question is in the expected state prior of booting from it, but nothing else. It is intended mount the ISO only after having been verified, then having the glue to call expected boot options from there. This means that in the stretched hypothesis that ext2/ext3/ext4/ partition is malformed to trigger a parsing vulnerability of Heads, the kexec call which will boot ISO’ OS’ initrd, which will then remount the ISO would also need to be compromised by that same malformity. Again highly improbable. Having a USB thumb drive firmware hacked in such way that it provides different blocks in case of gpg detached verification then when mounted is also highly improbable. Now, having a USB device exposing many other devices than a storage device will not be permitted by Heads, simply because those others USB drivers are not available.
I cannot think, right now, of a case where an infected/compromise USB thumb drive would infect Heads outside of a possible, while low probability, runtime exploitation that would vanish once Heads boots into the chainloaded OS after the kexec call. As opposed to preparing a USB boot drive, the ISO boot method is considered safer since a single bit corruption of the ISO would be detected prior of booting the ISO. As opposed to a burnded dvd/prepared single installer USB thumb drive, the USB Thumb drive containing the ISO can be untrusted, since the validation is made prior of booting the installation media.
Agreed that there is a small incongruity with the installation process, a chiken/egg problem that lies in the absence of /boot mountable filesystem available for HOTP to drop its /boot/kexec_hotp_counter, which is required under hotp board variants to provide its counter protection (guaranteeing a hotp enabled USB Security dongle can only be used on one device for the moment, and in sync between devices, as opposed to TOTP which counter is inside of the TPM and not requiring an OS to be installed to have firmware measurements sealed prior of OS installation.
Heads currently requires an installed OS to seal firmware measurements through HOTP into the USB Security dongle. This is why, if no OS is detected, Heads will propose to install one. And then, will propose to “OEM Factory Reset/Re-Ownership” on next reboot, when an OS is detected. Being paranoid, you could Upgrading Heads - Heads - Wiki to make sure that the firmware was not tampered by a compromised USB Thumb drive, while again, highly improbable.
thanks a lot for long explanation.
so, in conclusion,
ISO boot-able media, is a better way to install Qubes,
rather than using USB installation media,
because it does ISO verification.
Cannot USB boot from other media, non USB thumb drive,
because driver are not available in Heads.
Means, other devices, non USB thumb drives, also cannot infect Heads,
simply because not even can connect.
USB boot from external DVD is not necessary,
because firmware tampering, by compromised USB thumb drive,
is highly improbable, for below reasons:
during USB boot, Heads will auto verify Qubes ISO,
by using RSK stored under Heads,
& the detached PGP signature, stored side by side with the ISO.
so, in case of corrupted / compromised ISO, verification will fail, & so do USB boot,
means ISO mounting fail.
The attack surface of USB device, will require a combination of many things:
USB thumb drive tampered firmware, USB controller drivers inside Heads, Heads kernel’s filesystem drivers (ext2/ext3/ext4/Jolliet(ISO)/), USB thumb drive filesystem exploited bugs, busybox from the versions packed under flashed Heads version used, malformed ext2/ext3/ext4 partition, compromised kexec call. On top of that, if detached signature are validated, we add the gpg toolstack tools packed under Heads.
After OS installation, at each boot, Heads will verify the integrity of firmware,
which is expected to be consistent across boots, firmware tampering will be detected.
Heads is ram based recovery shell,
executed from a measured boot coreboot,
then continued with measured Heads’ modules,
which then will be replaced with loaded kernel+initrd+xen & OS.
Any runtime exploitation would vanish, once Heads boots into OS.
the only possible hypothesis is, but also highly improbable,
USB thumb drive firmware is compromised,
then give fake valid verification of,
compromised detached PGP signature & compromised ISO.
additional info: 29 types of USB attacks
is it possible to see/cat Qubes’ RSK, kept inside Heads, after it has been flashed into chip,
maybe via Heads recovery shell ?
You can see the distro keys (public keys) for OS currently known to distribute ISO detached signatures, under Heads recovery shell under
Here for Qubes 4.x distro signing key, as under github:
initrd/ is what is being commonly packed for all boards(scripts, configs and folders), and is the equivalent of
/ as seen under Heads recovery shell)
I did the weirdest way. I had a black screen problem. I used virtual box on windows.
I did not install QubesOS inside a virtual machine, but through a virtual machine. Instead of using a virtual disk, I created a fake vitrual disk with this command :
VBoxManage internalcommands createrawvmdk -filename "C:Users<user_name>VirtualBox VMs<VM_folder_name><file_name>.vmdk" -rawdisk .PhysicalDrive#
Where # is the drive wanted. I lunched the iso as a media and during installation, Qubes OS detected the empty free space on my disk and I choose the auto setup for mounting point. It worked perfectly. Now I have a independent QubesOS on its own crypted partition. Qubes is not running inside a virtual machine it was just for installation to disk. A nice trick.
I always installed it that way.
Even without thinking through this, i already liked it! Nice!
What I don’t like, you did it on Windows.
For me, the question of all questions would be: how to install it on a clean, stock hardware computer, without using any USB for any purpose.
Perhaps this might be a sufficiently good question to create a new topic.
You can avoid USB by using a CD drive. If you don’t have it, you could install Qubes on a different laptop that has it, and then you put the SSD into your new laptop.
Thanks. (Twice) “Actually” (for each of your paragraphs, haha)…
this passed (away) unnoticed… So I tried here with a different question… Still looks, until your post, I’m not clear enough with the subject point…
Should I say, I’m not surprised with your answer. Thanks once again for sanity check.