Today, QubesOS does not offer any protection against malware persistence, not even at dom0 or hypervisor level, as QubesOS completely lack boot verification. QubesOS does not even offer boot verification as an optional addition. The few desktop operating system that does implement some kind of verified boot, such as Ubuntu, only verifies kernel space, and fail to lock trust to only their own signing key, thus providing little to no security benefit in practice.
A few months back I set out on a mission to see if it is viable today for an OS vendor to implement full protection against malware persistence, relying only on functionality present in regular hardware and BIOS firmware, such as Secure Boot and TPM support. I decided to try to protect a custom built “live” version of a regular Linux distribution in this sense, so that all of user space including configuration is fully verified at boot, and below is a write-up of my findings, for anyone interesting in learning how it is done, or how to replicate it. Not only is it possible to do, but since at least a year back, all tooling exists to do so easily.
Every Linux distribution that is serious about security should implement this now, as it improves system security in very tangible ways. To the best of my knowledge, no Linux distribution at all implements full boot verification today, not even security focused ones like SecureBlue. What follows is how to turn it into a reality.
At the end is some reflections about how to also turn QubesOS hypervisor, dom0, sys-usb, sys-net, sys-firewall and sys-whonix into being fully immutable and fully verified at boot, in a secure way fully preventing malware persistence.
Overview:
Legacy boot will be disabled, and Secure Boot will be enabled in BIOS settings, with Deny Execution as default policy on violations for OpROMs and bootloaders alike. All default provisioned Secure Boot certificates will be deleted, and your own will be provisioned instead, plus hashes for all OpROMs logged to tpm2_eventlog during regular system startup. The latter is needed to ensure BIOS can load and execute the graphic driver from your GPU PCI-e slot, so you have visuals in early boot and still can access BIOS settings. This provides the foundation for ensuring only your own authentic OS images can be booted, nothing else.
The only step the user installing the operating system needs to do themselves is enabling Secure Boot and putting Secure Boot in Setup Mode by deleting all existing certificates through BIOS settings. The actual provisioning of your own OS signing key can be done by the installation program for your own OS. Once installed, the user never have to do anything again, everything will be transparent to them, as new OS images are signed by the same already enrolled Secure Boot key.
Secure Boot will now validate the EFI bootloader, and refuse to execute it unless it is signed by you. The EFI bootloader will be a Unified Kernel Image (UKI), containing the Linux kernel, the initrd, and the boot parameters. This ensures an attacker can not modify or replace the kernel nor initrd, nor modify the boot parameters in any way, as all of that is signed as a unit and verified at boot.
The whole of user space, including all apps and configuration, will be an immutable file system image (a single squashfs or erofs image file). A dm-verity file for the immutable image, containing all block hashes for the immutable image, will also be stored next to the image. Boot parameters will include the partition UUID and file path to the immutable image to boot. Boot parameters will also include a dm-verity root hash for the immutable image. The already verified authentic and unmodified initrd will read the already verified and authentic boot parameters. It will setup dm-verity in “reboot-on-corruption” mode, using the immutable image and its dm-verity file. Since the root hash is already verified authentic and unmodified, any modifications done to either the immutable image or dm-verity file will be detected when the corresponding block is being read, since dm-verity verifies all reads. The system will refuse to continue execution if this happens, making sure an attacker can not affect system execution in any way, not even by causing targetted I/O errors (dm-verity default mode). Since verification happens on each block read rather than full scan at boot, there is negligable boot time overhead, and any modifications happening post boot will also be detected. All persistence of malicious code is prevented.
To support writes, initrd will mount an overlay as “/”, with the dm-verity protected file system mount as lower layer, and a tmpfs (RAM backed) file system as upper writable layer. This ensures Linux functions as it should, without any support for persistence. If the user wants to save files or app data, they can create a separate LUKS encrypted partition, and store those files there.
An OS vendor would atomically replace the UKI image, file system image, and dm-verity file upon system update. This whole setup is very similar to the setup on Android devices, which also uses dm-verity in this manner. The OS vendor would likely also allow persisting user files and app data, for example by mounting the encrypted user data volume at a certain path at login, like in Tails, or as the home folder, like on Android.
Rollback protection to guard against downgrade attacks should also be possible to implement. One can apparently setup UEFI variables such that only a verified bootloader can update the value, making it possible to write a counter that way, and making all UKIs ever signed by the same Secure Boot key check what the counter is and refuse to boot if counter is higher than their built-in value, preventing downgrade attacks. This solution is apparently implemented by Secure Version Number (SVN, Windows) and Secure Boot Advanced Targeting (SBAT, Linux). An OS vendor would want to do this. I haven’t made a proof-of-concept of this. If you are just making your own OS images for yourself, you can just enroll a new Secure Boot key each time. But an OS vendor would want to implement rollback protection, so a malware cannot rollback to earlier more vulnerable versions.
In addition to preventing malware persistence, one might want to protect against attackers with physical access to your device, or allow for attesting system security post-install. Setting a BIOS boot and setup password raises the bar a little against a physical attacker (Evil Maid) trying to disable Secure Boot or replace the enrolled keys with their own. You would notice any fool play even if they jumper reset your BIOS, as long as the attacker doesn’t know your BIOS password and thus cannot set the right one again. This should lower the risk you leak your disk encryption passphrase to them. The attacker cannot replace the BIOS, since it is verified by keys burned into fuses on your device. But they might be able to change specific BIOS settings by reflashing the chip holding them, and they definitely can get your disk encryption passphrase using a pinhole camera or hardware keylogger anyway.
As for attestation, no BIOS today prints the hash of the enrolled Secure Boot key during boot, making it impossible to inspect from a verified environment whether the next step that will be loaded is also secure. Theoretically, values written to TPM PCR registers can be used for attestation, but it is unclear if they are adequately protected. At least one BIOS vendor (MSI) is also adding that Secure Boot is enabled to the PCR registers, despite it being effectively disabled by execution policies being set to “Always execute”. Ideally, the BIOS would print the full hash of the enrolled signing key at boot, for you to verify, or at least not outright lie to the TPM about system state.
Detailed how-to instructions:
Step 1 - Install and setup Linux distribution exactly like you want it
Start by installing a regular Linux distribution in the usual way. I used Linux Mint, so this should work just fine on any Debian based distribution. Install any software you might want, and make any configuration changes you might want. If installing proprietary NVidia drivers, make sure to install the version with properly signed kernel modules, or they won’t load. On Linux Mint, do this by install linux-modules-nvidia-VER-generic before nvidia-driver-VER. To reduce image size, remove unnecessary application, all kernels except the latest, and clear package manager cache and similar. Do not access any privacy sensitive files nor enter any personal passwords at all during this step. The image you build must be made in such a way you could distribute it to anyone.
Step 2 - Prepare initrd to support booting dm-verity protected immutable image
Add support for dm-verity in the initrd image. This will be used to allow initrd to mount the user space image in such a way that system can detect any modification made and prevent further execution of the system. Create “/usr/share/initramfs-tools/hooks/veritysetup” with the following content:
#!/bin/sh
. /usr/share/initramfs-tools/hook-functions
copy_exec /sbin/veritysetup /sbin
manual_add_modules dm_verity
Edit “/usr/share/initramfs-tools/init”. Add the following two code snippets in the respective correct list in that file, to allow passing in our custom boot parameters.
export SQUASHFILE=
export VERITYHASH=
squashfile=*)
SQUASHFILE=${x#squashfile=}
;;
verityhash=*)
VERITYHASH=${x#verityhash=}
;;
Finally also edit “/usr/share/initramfs-tools/scripts/local” to add support for mounting dm-verity protected immutable file system images, replacing the support for the regular root file system mounting.
Replace this:
# Mount root
# shellcheck disable=SC2086
mount ${roflag} ${FSTYPE:+-t "${FSTYPE}"} ${ROOTFLAGS} "${ROOT}" "${rootmnt?}"
With this:
# Mount read-only file system image
modprobe ext4
modprobe squashfs
modprobe dm_verity
mkdir -p /fsimage
mkdir -p /medium
mount -t ext4 -o ro,noatime "$ROOT" /medium
veritysetup open --reboot-on-corruption "/medium/${SQUASHFILE#/}" verityimg "/medium/${SQUASHFILE#/}.verity" "$VERITYHASH"
mount -t squashfs -o ro,noatime /dev/mapper/verityimg /fsimage
# Mount writable upper layer
mkdir -p /cow
mount -t tmpfs -o rw,noatime,mode=0755 tmpfs /cow
# Mount the combination of both as the root
modprobe overlayfs
mkdir -p /cow/upper
mkdir -p /cow/work
mount -t overlay -o noatime,lowerdir=/fsimage,upperdir=/cow/upper,workdir=/cow/work overlay "$rootmnt"
# Make the underlying filesystems visible inside
mkdir -p "$rootmnt/livemnt/medium"
mkdir -p "$rootmnt/livemnt/fsimage"
mkdir -p "$rootmnt/livemnt/cow"
mount -o bind /medium "$rootmnt/livemnt/medium"
mount -o bind /fsimage "$rootmnt/livemnt/fsimage"
mount -o bind /cow "$rootmnt/livemnt/cow"
Now run the below as root to regenerate the “initrd.img” file. Run this from within the installation itself.
chmod +x /usr/share/initramfs-tools/hooks/veritysetup
update-initramfs -u
Edit /etc/fstab so it does not mount and even mention any file systems. The user space must not auto-mount any file systems ever.
Step 3 - Convert into a dm-verity protected immutable OS image
Create an ext4 partition to hold the immutable file system images, and nothing else. Place your terminal in that partition, and execute the below as root. Here I use squashfs, but erofs is recommended today because of better file system attribute support. Execute this from another Linux installation to ensure no files are updated while the image generation progresses.
mksquashfs /path/to/filesystem-root filesystem.squashfs -noappend
veritysetup format filesystem.squashfs filesystem.squashfs.verity
Take note of the root hash mentioned in that last command. This is the one we will need to add to our boot parameters in the UKI image. Now create the UKI image as root. Replace ${UUID} with the UUID of the ext4 partition holding the squashfs image and dm-verity file, and ${ROOTHASH} with the actual root hash value.
ukify build --linux /path/to/filesystem-root/boot/vmlinuz --initrd /path/to/filesystem-root/boot/initrd.img --cmdline "root=UUID=${UUID} squashfile=/filesystem.squashfs verityhash=${ROOTHASH} ro quiet splash"
It is advisable to add more security hardening boot options, including zero-on-free support to prevent cold boot attacks, and disabling of EFI pstore and ERST to prevent persisting system logs to EFI or ACPI variables, which can leak information about files in encrypted volumes. I would add this after the “splash” option: “slab_nomerge slub_debug=FZ mce=0 vsyscall=none init_on_free=1 mds=full,nosmt page_alloc.shuffle=1 randomize_kstack_offset=on efi_pstore.pstore_disable=1 erst_disable spec_store_bypass_disable=on”, which is the boot parameters Tails is using. I have not verified whether they work on Ubuntu kernels though.
Delete any data already existing on the EFI system partition, and then place the generated UKI as “/EFI/BOOT/BOOTX64.EFI”. This should be enough, but just to be certain, use “efibootmgr” to delete all existing configurations, and then register the new one:
efibootmgr --create --disk /dev/sda --part 1 --label "My operating system" --loader '\EFI\BOOT\BOOTX64.EFI'
Step 4 - Sign UKI image and enroll keys to Secure Boot.
Make sure Secure Boot is in Setup Mode by deleting all enrolled certificates from the BIOS settings before booting. Secure Boot should be enabled in BIOS settings, but will present as disabled during boot since in Setup Mode.
Now generate a new signing key and sign the UKI image. Make sure you do not save the generated keys anywhere where an attacker can later access them. If you are an OS vendor, you should adequately protect them on some hardware security module on some offline system. If you are just creating this OS images for yourself, you can just write the keys to a RAM backed file system while offline on a trusted system, so they aren’t persisted at all. You won’t need to sign anything more in the future, as you can just put Secure Boot back in Setup Mode and enroll new keys each time.
openssl req -newkey rsa:4096 -nodes -keyout sign.key -new -x509 -sha256 -days 3650 -subj "/CN=My EFI Signing Key/" -out sign.crt
sbsign --key sign.key --cert sign.crt --output /EFI/BOOT/BOOTX64.EFI /EFI/BOOT/BOOTX64.EFI
Now, fetch all hash values for OpROM images loaded at boot. On some systems, like laptops, there will be none. If you have a desktop computer with a discrete GPU, there will usually be one OpROM hash for the GPU. If you have a RAID controller, there might be one more. Take note of each SHA256 hash for the EV_EFI_BOOT_SERVICES_DRIVER entries that was measured to the TPM2 PCR registers at boot.
tpm2_eventlog /sys/kernel/security/tpm0/binary_bios_measurements
An OS vendor might want to fetch all OpROM hashes for systems they want to support, and add them to a machine specific db, signed by the OS vendors PK and KEK keys. It might be tempting to just fetch the OpROM hashes during the installation program, but that would mean the PK and KEK keys needs to be generated by the installation program, and there is no way to attest they were generated securily and destroyed securely, so that would break the ability to attest system authencity. The OS vendor needs to take care to only sign OpROM hashes they themselves have verified authentic.
Now, create and sign the PK, KEK and db EFI variable data.
GUID=$(uuidgen --random)
cert-to-efi-sig-list -g $GUID sign.crt sign.esl
echo ${OPROM_HASH} | xxd -r -p > hash
sbsiglist --owner $GUID --type sha256 --output oprom.esl hash
cat sign.esl oprom.esl > db.esl
openssl req -newkey rsa:4096 -nodes -keyout pk.key -new -x509 -sha256 -days 3650 -subj "/CN=My Platform Key/" -out pk.crt
openssl req -newkey rsa:4096 -nodes -keyout kek.key -new -x509 -sha256 -days 3650 -subj "/CN=My Key Exchange Key/" -out kek.crt
cert-to-efi-sig-list -g $GUID pk.crt pk.esl
cert-to-efi-sig-list -g $GUID kek.crt kek.esl
sign-efi-sig-list -g $GUID -k pk.key -c pk.crt PK pk.esl pk.auth
sign-efi-sig-list -g $GUID -k pk.key -c pk.crt KEK kek.esl kek.auth
sign-efi-sig-list -g $GUID -k kek.key -c kek.crt db db.esl db.auth
Upload it to the EFI variable storage as root. The last command below uploads the PK variable, which will cause Secure Boot to be enabled, and Setup Mode to be exited. The next reboot, Secure Boot will be enforced.
mkdir a
mkdir a/PK
mkdir a/KEK
mkdir a/db
mv pk.auth a/PK/PK.auth
mv kek.auth a/KEK/KEK.auth
mv db.auth a/db/db.auth
sbkeysync --keystore a --verbose
sbkeysync --keystore a --verbose --pk
That is all. An OS vendor would create and sign all EFI variable data in their own secure environment, distribute the signed data (.auth files) in their installation program, which uploads them to the EFI variable storage on the user’s machine.
A fully installed system will have uploaded the PK, KEK and db EFI variables to EFI variable storage, will have a EFI system partition holding a single file “/EFI/BOOT/BOOTX64.EFI”, and will have an ext4 partition holding two files “filesystem.squashfs” and “filesystem.squashfs.verity”. That is all. The remaining disk space can be allocated to one or more user data partition, all preferably LUKS encrypted. On top of that, the user is supposed to have enabled Secure Boot, set default violation policy to Deny Execution for everything, and have deleted all pre-provisioned certificates to set Secure Boot in Setup Mode, prior to running the installation program.
Future:
For the future, a quick internet search reveals that Xen has full support for being bundled in a UKI like EFI image, together with all Xen configuration, dom0 kernel and dom0 initrd. This alone should be enough to implement full boot verification in QubesOS in a secure way, that is easy to use, and works on almost all hardware. This would remove the possibility for the user to modify dom0 configuration, but the user is not supposed to do that anyway, and if the user can, so might an attacker. It’s better to be certain the OS including all of dom0 is in an authentic and secure state.
Individual template qubes and app qubes will not be protected this way though, but the isolation between them will. QubesOS design is fundamentally incompatible with extending verification further in, as apps would need to be immutable and signed images too for that to work. Maybe Flatpaks are one possible avenue for the future, to allow verifying the full system authencity.
Any criticism or thoughts are welcome!