Install Qubes OS with boot partition and a detached LUKS header on USB

The purpose of having a detached LUKS header and boot partition on a separate external disk is to achieve a deniable encryption:
Deniable encryption - Wikipedia
The encrypted disk will look like an unused/empty unpartitioned disk.

Boot into Qubes OS installer and on GUI screen switch to shell on another TTY by pressing Ctrl+Alt+F2.

Assuming:
/dev/sda - disk where you want to install Qubes OS
/dev/sdb - USB disk where you want to install Qubes OS boot partition

Find out the right disk names on your machine in Qubes OS installer shell by running this command and checking the output:

fdisk -l

In this shell run these commands:

dd if=/dev/zero of=header.img bs=16M count=1
cryptsetup luksFormat /dev/sda -c aes-xts-plain64 -h sha512 -s 512 -y -i 10000 --use-random --force-password --header header.img
cryptsetup open --header header.img /dev/sda luks
mkfs.btrfs --csum xxhash -L qubes_dom0 -d single /dev/mapper/luks

Return to Qubes OS installer GUI by pressing Ctrl+Alt+F6.
Configure the installation as normal except for Installation Destination:
Installation guide | Qubes OS
At the Installation Destination screen click on “Refresh
” at the bottom right corner and in the opened window press on “Disk rescan” button and then OK.
At the Device Selection choose your Qubes OS installation destination and boot partition destination disks /dev/sda and /dev/sdb.
At the Storage Configuration select “Advanced Custom (Blivet-GUI)”.
Press Done.
In the “BLIVET GUI PARTITIONING” screen select “sdb” disk.
Delete the old partitions on the disk if needed.

Add new partition using “+” button:
Device type: Partition
Size: 512 MiB
Filesystem: EFI System
Mountpoint: /boot/efi
Press OK button.

Add new partition using “+” button:
Device type: Partition
Size: 1 GiB
Filesystem: ext4
Mountpoint: /boot
Press OK button.

In the “BLIVET GUI PARTITIONING” screen select “qubes_dom0” Btrfs Volume.
Add new subvolume using “+” button:
Name: root
Mountpoint: /
Press OK button.

Press on Done button in “BLIVET GUI PARTITIONING” and then “Accept Changes” button in “SUMMARY OF CHANGES” window.
Press “Begin Installation” button.

After installation is completed don’t press “Reboot System” button.
Switch to shell on another TTY by pressing Ctrl+Alt+F2.
Run these commands in the installer shell:

cp header.img /mnt/sysroot/root/
cd /mnt/sysroot
mount -t proc /proc proc/
mount -t sysfs /sys sys/
mount --rbind /dev dev/
chroot /mnt/sysroot
btrfs subvolume create /swap
btrfs filesystem mkswapfile --size=4g --uuid clear /swap/swapfile

Edit /etc/fstab file using nano or any other text editor:

nano /etc/fstab

Add noauto option to the /boot and /boot/efi mounts like this:

UUID=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX /boot                   ext4    defaults,discard,noauto 1 2
UUID=XXXX-XXXX          /boot/efi               vfat    umask=0077,shortname=winnt,discard,noauto 0 2

And add this line at the end:

/swap/swapfile none swap defaults,discard=once 0 0

Save and close /etc/fstab file.
Edit /etc/default/grub file using nano or any other text editor:

nano /etc/default/grub

Remove rd.luks.uuid=XXXXX option and add rd.driver.pre=btrfs option to the GRUB_CMDLINE_LINUX variable. It should look like this:

GRUB_CMDLINE_LINUX="rd.driver.pre=btrfs plymouth.ignore-serial-consoles 6.7.7-1.qubes.fc37.x86_64 x86_64 rhgb quiet"

Save and close /etc/default/grub file.
Create new /etc/dracut.conf.d/crypt.conf file using nano or any other text editor:

nano /etc/dracut.conf.d/crypt.conf

Add this text inside:

add_dracutmodules+=" crypt "
install_items+=" /root/header.img "

Save and close /etc/dracut.conf.d/crypt.conf file.
Find out the ID of your Qubes OS destination disk /dev/sda:

ls -la /dev/disk/by-id/ | grep /sda

For example it’ll look like this:

lrwxrwxrwx 1 root root   9 May 10 20:15 ata-YOUR_DISK_ID -> ../../sda

Here ata-YOUR_DISK_ID is your disk ID.

Edit /etc/crypttab file using nano or any other text editor:

nano /etc/crypttab

Add this text inside:

luks /dev/disk/by-id/ata-YOUR_DISK_ID none header=/root/header.img,force,discard

Save and close /etc/crypttab file.
Run these commands:

grub2-mkconfig -o /boot/grub2/grub.cfg
dracut -f --regenerate-all
exit

Return to Qubes OS installer GUI by pressing Ctrl+Alt+F6.
Press on “Reboot System” button.
After this proceed with normal Qubes OS post-installation process:
Installation guide | Qubes OS

Since /boot and /boot/efi partitions are stored on USB disk then you’ll need to attach this disk to your dom0 when doing dom0 updates so the files there will be updated.
It’s better to use disposable sys-usb for this setup.
When you want to update your dom0 you can follow these steps:
Disconnect all your USB devices.
Restart sys-usb to clear it’s state.
Connect your USB disk with Qubes OS /boot partition.
Run this command in dom0 to mount the /boot and /boot/efi partitions in dom0 assuming that /dev/sda is the name of your USB disk with Qubes OS /boot partition in sys-usb:

qvm-block attach dom0 sys-usb:sda1 && qvm-block attach dom0 sys-usb:sda2 && sudo mount /boot && sudo mount /boot/efi

Run dom0 update.
After dom0 update is finished run this command in dom0 to unmount and remove /boot and /boot/efi partitions from dom0:

sudo umount /boot/efi /boot && qvm-block d dom0 sys-usb:sda1 && qvm-block d dom0 sys-usb:sda2

At this point you can disconnect the USB disk with Qubes OS /boot partition from your machine and continue to use sys-usb with other USB devices as normal.

NOTE

Since TRIM is enabled by default:
Disk Trimming
This could indicate that this disk is not unused and this could break the plausible deniability:
dm-crypt/Specialties - ArchWiki
So you may want to disable the TRIM, but this will reduce the disk performance.

7 Likes

Assuming the installer and the boot process itself have no USB protection (like sys-usb or USBGuard), how does one trust a USB device with complex proprietary firmware (such as USB storage) at install or boot time? I.e. how and why does one trust such a device at such early and vulnerable phase? I don’t dare to connect even the USB port to the UPS :slight_smile:

You can use USB device if it’s OK for your threat model.
USB device that you use to boot the Qubes OS from should be assumed as trusted.
The same way that USB device containing the Qubes OS installer that is used to install the Qubes OS should be assumed as trusted.
Installation security | Qubes OS
Instead of using USB device to boot Qubes OS from it could be considered to use the SD card reader (if available in your machine) and SD card to boot from. There is no firmware in SD cards (AFAIK).

I was wrong, there is a firmware there as well:
https://www.bunniestudios.com/blog/2013/on-hacking-microsd-cards/

So I guess if you use a separate PCI USB controller or PCI SD card reader only to boot from your boot device then it’ll be the same as booting from SATA/NVMe drive.
You’ll have to trust that your boot device firmware is not malicious the same way that you’ll need to trust that your SATA/NVMe drive firmware is not malicious. And that both your boot device and SATA/NVMe drive couldn’t be physically accessed by attacker.

Isn’t the point that with one’s USB goes access to the physical device? (Assuming one carries the USB once shutting down the device and walking away.)

There is always a chance that attacker can access it in certain situations (like during border control search or when you sleep). The same goes for attacker accessing the machine.
Also USB device firmware could be malicious even before it’ll end up in your hands. But the same goes for any other hardware.

I think the main issue with USB storage specifically is that it can behave as other devices like USB keyboard.
On the other hand even if e.g. SD card has malicious firmware then it still can’t be used like a keyboard. It only can be used to further exploit the vulnerability in the SD card reader to gain further access of your system.

@apparatus

I know the things you mention but they still don’t answer the actual question. However, since it requires a more in-depth discussion, it would better fit a separate thread :slight_smile:

I’ve created separate topic:

Very nice topic. I had a similar config long ago. With the difference that it was on a Desktop which booted via PXE by default. The tftp and early boot stage server was via an SBC with wireless connection to router connected to the desktop via Ethernet. The SBC was hidden inside air duct. And the LUKS volume started right after a dummy Windows partition. So without the PXE server (e.g. if the PC was stolen and moved elsewhere), PC would have booted to Windows and anyone would have thought it to be an ordinary Windows PC.

There was an independent replica of bootloader and /boot which I keep on a USB stick in a place no one would guess.

@apparatus: Thank you for sharing. Could you please explain in a little bit more depth what you are doing or recommending exactly?

To my understanding
a) you are using a btrfs volume instead or additionally to an lvm volume? why?
b) you are placing the boot partition and the luks header of the root device on an usb stick?
c) you are setting up a swapfile? why?
d) you are passing this changes to grub and dracut

Usually it is not possible to umount /boot during runtime of the OS. But it looks like your setups unmounts /boot after startup and the user can detach the usb stick - which could be considered a security feature. At least you are writing that the user has to reattach the usb stick and /boot if one wants to update kernel, modules and initramfs.

The reasons for using btrfs instead of lvm can be found here:

Also btrfs is now default filesystem in Fedora instead of lvm:
https://fedoraproject.org/wiki/Changes/BtrfsByDefault

But you can use this guide and change btrfs to lvm as well.

It’s to achieve deniable encryption:

Without USB stick the disk will look like unused/empty unpartitioned disk.

You mean why use swap at all or why swap file instead of separate partition?
Swap is needed for dom0 not to fail in case of some rare memory exhausting situation.
I’m using swap file instead of separate swap partition because separate swap partition requires the disk to be partitioned and it’ll interfere with deniable encryption.
I’ve thought of a way to use separate swap partition without partitioning the disk, but it’s not supported by systemd:

Can you describe your question in more details? I didn’t get it.

You can safely unmount the /boot during runtime, the USB stick is only needed to boot the initramfs, you can remove the USB stick after you enter the disk password and dom0 will start to boot.
And if you have sys-usb then USB stick won’t be present in dom0 by default because all USB controllers will be attached to sys-usb.
You only need to mount the /boot and /boot/efi when you update dom0 system:

2 Likes

Thank you for adding these explanations. If I understood you correctly your setup creates a block device /dev/sda with encrypted data which provides more or less plausibly deniability. The famous wrench taken out of the equation. Does /dev/sda hold a partition table?

In my eyes your setup also adds another layer of security as the boot partition is a weak spot in all linux distros. Most distros don’t encrypt the boot partition and don’t cryptographically sign the files on the boot partition. It’s just the kernel, the modules, the initramfs and the EFI boot blobs, so what could possible go wrong? :wink:

That was just my thought process on

Looks like I can answer this question myself:

It does not expose a partition table.

Awesome!

Yes, it’ll be a /dev/sda device without partition table.
But since TRIM is enabled there may be some security implications:

Disk Trimming
So you may want to disable TRIM.

Yes, it can protect the /boot and /boot/efi from malicious modifications in some cases.
But if you’re using USB disk with the same USB controller attached to sys-usb that you use to connect some untrusted USB devices then it’s possible that some malicious USB device will compromise sys-usb, write malicious firmware in USB controller and restarting disposable sys-usb won’t get rid of it and this malware in the USB controller firmware may then modify the files on your boot USB disk that you connect to it.
Also unlike Anti evil maid or TrenchBoot, it won’t protect against malicious EFI firmware.

I’m stuck at this step:

After pressing the Done button, I see this error:

Error checking storage configuration. Click for details or press Done again to continue.
The existing unlocked LUKS device sda cannot be used for the installation without an encryption key specified for this device. Please, rescan the storage.

When clicking Close and then Done a second time, the Begin Installation button is greyed out.

I found this article after searching for the error:
https://linuxconfig.org/how-to-install-fedora-rhel-centos-via-kickstart-on-an-existing-luks-device

I tried cryptsetup close luks from the TTY, but couldn’t unlock the drive from the Advanced Blivet GUI since there is no option to specify that the header is detached.

I can think of several ways to proceed, but I’m not sure if they will work or how hard and time consuming they would be:

  1. Create an updates.img file like in the tutorial above, where we comment the verify_unlocked_devices_have_key line.
    As far as I understood, if we want to put it on USB drive, we should follow these steps:
    dd if=updates.img of=/dev/sdb
    updates.img should be formatted with ext2.
    Then, we can append inst.updates=/dev/sdb to the kernel command line in the beginning of the installation.
    I think we can use the same USB drive that we will later put the detached /boot and LUKS headers on, because we only need inst.updates from it in the beginning, before we partition it.

  2. Install Qubes not by clicking Begin Installation, but by using the CLI. This seems to involve formatting the drives, copying a bunch of files from the installation media and applying the changes we made in the GUI for keybaord layout, timezone, user and so on.

  3. Use KickStart (the scripts to automate Fedora installations) somehow.

  4. Patch Anaconda to use cryptsetup option --header header.img when unlocking from the GUI via the inst.updates method. Then, before clicking Begin Installation, close the luks volume from the TTY and open it via the GUI.

  5. Install without detached LUKS header. After the installation, move the header to the USB drive. More specifically, to the initramfs via dracut, right?

I’m quite confused, to be honest. I’d appreciate any help. In the meantime I’ll try to solve it myself.

Finally, a word of advice I should have followed: When trying to follow a complex tutorial for the first time, use a simple LUKS password like 123 to avoid typing the real complex password over and over.

I didn’t have this error at the time of writing this guide.
Did you follow all the steps and used the same commands without your custom modifications?
I’ll try to check this again using the Qubes OS 4.2.3 installer ISO in a VM for a test.

Thanks for checking!

I followed the steps without any modifications whatsoever. I have a txt file where I wrote down everything I did, copying each step from the guide and double checking it before running it.

From previous installs of Qubes without a detached /boot or LUKS header, I have a vague recollection that after setting up LUKS in the TTY (just to increase iter-time, I think), I had to cryptsetup close the from the TTY before using the GUI. Though I didn’t get that error before, it was something different. It was long ago, so I don’t remember.

I’ve just tried it with Qubes OS 4.2.3 installer ISO in a qube and it worked for me without this error.
Can you try it in a VM to check that you’ve doing everything according to the guide and it’s not some hardware-specific error?
If you’ll try it in a qube then you can switch TTY by running this command in dom0:

xdotool selectwindow key ctrl+alt+F2

and

xdotool selectwindow key ctrl+alt+F6

And then clicking the X-marked cursor on the HVM window.

1 Like

I tried it again and I didn’t get the error.

But I feel like Anaconda may be a bit unpredictable at times since the first time I also followed the instructions correctly. Is it possible that the previous contents of the USB drives or the HDD might have affected something?

Anyway, this is what I went through:
After I got the error that I tried to make the updates.img, which is just a cpio of the modified file
usr/lib64/python3.11/site-packages/pyanaconda/modules/storage/checker/utils.py.
It’s a different file from the one in the link I posted, but it’s the only one in the anaconda that has the line with verify_unlocked_devices_have_key. I guess they moved it at some point.
The docs say that to use it from a USB drive, it has to be in the ext2 format, not the cpio. But the tools only make a cpio file.
I extracted the updates.img file I made to see that it just contains the changed file along with the path.
On a dispVM I ran (truncated):

git clone --depth 1 https://github.com/rhinstaller/anaconda -b f37-release
cd anaconda
grep -rnli verify_unlock *
vi pyanaconda/modules/storage/checker/utils.py
./scripts/makeupdates
gzip -dc updates.img | cpio -idu
sudo apt install -y util-linux
sudo dd if=/dev/zero of=updates.img bs=128k count=1
sudo mkfs.ext2 updates.img 
sudo mount updates.img /mnt
sudo mkdir -p /mnt/usr/lib64/python3.11/site-packages/pyanaconda/modules/storage/checker
sudo cp usr/lib64/python3.11/site-packages/pyanaconda/modules/storage/checker/utils.py /mnt/usr/lib64/python3.11/site-packages/pyanaconda/modules/storage/checker/

Then I unmounted and dd’d the updates.img file to the second USB drive (not the one with the installer). I appended inst.updates=/dev/sdb to the kernel command line of the installer when, after quiet, but it didn’t work - probably the 2 USB drives got renumbered since I didn’t use by-uuid.

Instead, I went and commented the line directly after booting the installer just to see if it would work. But then I wasn’t able to select EFI System Partition for /boot/efi on the USB drive for some reason. It asked me to first create a partition table. Neither msdos, nor gpt showed me the EFI System Partition option in the GUI. I tried several times to dd if=/dev/zero the beginning of the USB drive, then to rescan the disks from the GUI, but still no EFI System Partition option after I had to select a partition table from the GUI. I tried to make both types of tables one after the other with parted and fdisk, just to see if the GUI would show me different options, but it didn’t. I even made the EFI partition with fdisk, but then the GUI just showed ext4 when I tried to modify the partition. I even rebooted once or twice between the different tries.

I rebooted again and ran the cryptsetup and mkfs.btrfs commands on /dev/sda again, but then the GUI, after rescanning the disks, didn’t show the btrfs Volume of /dev/sda this time.

I rebooted a few times between all the failed attempts, but it didn’t help. It can’t be related to the commented out line since it only affected the current boot and I only did it once. After much frustration I then left the laptop turned off for a few minutes and booted again, and voila, it worked on the first try without an error, with the EFI System Partition option available and with the btrfs volume showing up. Each time I followed the same 4 commands (from dd to mkfs.btrfs), but the GUI was acting unpredictably.

Not sure if shutting down for more than a few seconds did it, or it was luck, or if I have a failing HDD or USB drive. I am confident I didn’t mistype the commands. And I went for the parted and fdisk commands only after the GUI mysteriously didn’t show the EFI System Partition option.

Has anyone noticed similar inconsistent behavior on Anaconda?
Why the error the first time, but not now?
Why the missing EFI System Partition option?
Why the missing btrfs volume?

Thanks again for testing, by the way.

(Anyway, I don’t want to derail the thread since it worked in the end (at least so far, it’s still installing). It could be a number of reasons for the issues I saw, but they don’t seem directly related to the topic at hand. Thank you for the guide and sorry for the trouble. :slight_smile: )