Install Qubes OS with boot partition and a detached LUKS header on USB

Community Guides
61

Hi. Actually no. I do not need this much security measures at this moment. But if I ever do implement a similar feature, for sure I will write a guide on this.

62

I used the setup from this guide several times with no problems in the past, but in 4.3 RC2 it somehow didn’t work this time. Somehow after finishing the setup and rebooting the system, I got a boot failure.
Not sure if something changed in the new release or if I did something wrong this time, but essentially I had to put these lines in /etc/dracut.conf.d/crypt.conf:

add_dracutmodules+=" crypt "
install_items+=" /root/header.img /usr/sbin/cryptsetup /etc/crypttab "

And in /etc/crypttab:

luks /dev/disk/by-id/ata-YOUR_DISK_ID none header=/root/header.img,force,x-initrd.attach,discard

Then it worked again.

Additionally, because of the new policy system in 4.3, I wasn’t able to attach my USB to dom0 anymore. I tried to add a policy in the Qubes Policy Editor:

30-allow-dom0-block

qubes.BlockList * dom0 sys-usb allow
qubes.BlockAttach * dom0 sys-usb allow
qubes.BlockDetach * dom0 sys-usb allow

But somehow that didn’t work. Not sure if there is a bug or I made a mistake. But for now I helped myself by updating with this command:

qvm-run -u root sys-usb 'umount -f /dev/sdX1 2>/dev/null || true; umount -f /dev/sdX2 2>/dev/null || true; mkdir -p /mnt/boot/efi; mount /dev/sdX2 /mnt/boot; mount /dev/sdX1 /mnt/boot/efi' && sudo qubes-dom0-update && sudo tar -C /boot -cpf - . | qvm-run --pass-io sys-usb 'sudo tar -C /mnt/boot -xpf -' && qvm-run -u root sys-usb 'sync; umount /mnt/boot/efi /mnt/boot'`
2 Likes
Qubes 4.3: USB Stick suddenly started plugging-in in sys-usb only in storage mode. Usb_modeswitch doesn't help
Can't install Qubes on new machine - kernel panic with latest kernel & lightdm errors on boot
Qubes 4.3: USB Stick suddenly started plugging-in in sys-usb only in storage mode. Usb_modeswitch doesn't help
Qubes 4.3: USB Stick suddenly started plugging-in in sys-usb only in storage mode. Usb_modeswitch doesn't help
63

Same here, the guide is still golden for 4.2, but not for 4.3

I had all the issues that @merowing mentioned. I think there’s something wrong with “getting password for disk encryption”, and my keyboard didn’t work (it works in grub and dracut emergency shell).

Luckily the update tool automatically created a LVM snapshot, and I was able to revert to it.

1 Like
Qubes 4.3: USB Stick suddenly started plugging-in in sys-usb only in storage mode. Usb_modeswitch doesn't help
64

I did a bit more research, my findings:

  • I read that when systemd modules is enabled in dracut, it uses systemd-cryptsetup, which is an independent implementation from cryptsetup, so it is natural that /usr/sbin/cryptsetup is not enabled. (I observed that systemd has been running in the dracut emergency shell)
  • Supposedly dracut should pick up entries from /etc/crypttab.initramfs and generate /etc/crypttab for initramfs. Maybe adding an entriy in /etc/crypttab.initramfs might help.
  • Since systemd is enabled, maybe we can try systemd-cryptsetup-genertor parameters instead of /etc/crypttab

Other rumors: (I have not tested them):

  • Enable rd.hostonly
  • Make sure the first line of /etc/crypttab is not comment
  • Add an entry of /
1 Like
65

I have this dracut config, and it is working:

  • 10-nvme.conf, content: add_drivers+=" nvme "
  • crypt.conf, content: add_dracutmodules+=" crypt "
  • 15-compress.conf, content: compress=“zstd -1”
  • lvm.conf, content: add_dracutmodules+=" lvm "
  • 99-hostonly.conf, content: hostonly=“no”
  • 10-crypt.conf, content: install_items+=" /etc/keys/key_dom0 /etc/crypttab /etc/keys/header_dom0 "
1 Like
66

I checked my working dracut config again and it actually looks like this:

add_dracutmodules+=" crypt "
install_items+=" /root/header.img /usr/sbin/cryptsetup /etc/crypttab "
hostonly=no

So this and the entry x-initrd.attach in /etc/crypttab are the only things I changed from the original guide, but I don’t know which additional entry actually caused it to work for me.

If I had to guess, I would say it’s the entry /etc/crypttab. I’ve read that dracut doesn’t always reliably include crypttab automatically in all setups. Therefore it’s probably more reliable to include it explicitly. /usr/sbin/cryptsetup is not necessary I guess, but it doesn’t hurt either.

But using systemd-cryptsetup-generator parameters is probably also a good alternative.

1 Like
Qubes 4.3: USB Stick suddenly started plugging-in in sys-usb only in storage mode. Usb_modeswitch doesn't help
Qubes 4.3: USB Stick suddenly started plugging-in in sys-usb only in storage mode. Usb_modeswitch doesn't help
Can't install Qubes on new machine - kernel panic with latest kernel & lightdm errors on boot
Qubes 4.3: USB Stick suddenly started plugging-in in sys-usb only in storage mode. Usb_modeswitch doesn't help
How to edit dracut.conf.d/crypt.conf after installation with external boot partition?
67

I found a working config for me: simply changing crypt to systemd-cryptsetup, no need to include other files.

Relevant bugs:

Basically it’s what I suspected, that there’s a bug where crypt module doesn’t work if systemd is enabled.

I’ve been also wondering why we have to manually add crypt or systemd-cryptsetup. After studying the code, my theory is the detached header prevents dracut from automatically detecting the filesystem type.

@merowing I’m not sure why your config works for you but not for me. I’d guess either you don’t have systemd module enabled, or you have some other LUKS encrypted devices (without detached header), which makes dracut to include systemd-cryptsetup. Perhaps you can verify using lsinitrd /boot/initramfs-...

2 Likes
Unable to type luks password after fresh 4.3.0 install with detached header
68

I found a workaround for the USB issue: Cannot attach `sys-usb` devices to dom0 · Issue #10361 · QubesOS/qubes-issues · GitHub

2 Likes
69

I think hostonly=no also affects the decision. Perhaps you can verify systemd-cryptsetup module is included.

1 Like
70

Thanks for the USB workaround! That helped.

Well I don’t have any other LUKS devices.
If I understand correctly, you upgraded from 4.2 to 4.3 right? Because I did a clean install of 4.3. Maybe that changed something? I don’t know. But it’s good that it worked out for you now too.

This is what lsinitrd -m /boot/initramfs-$(uname -r).img | egrep -i 'crypt|systemd' is showing:

systemd 
fips-crypto-policies 
systemd-ask-password 
systemd-initrd 
systemd-journald 
systemd-modules-load 
systemd-pcrphase 
systemd-sysctl 
systemd-sysusers 
systemd-tmpfiles 
systemd-udevd 
crypt 
systemd-cryptsetup 
dracut-systemd
1 Like
71

I see, so systemd-cryptsetup is also enabled in your setup. I don’t know what triggered it, but it confirms my theory (that crypt itself doesn’t work when systemd is enabled).

1 Like
72

I’m having trouble figuring out which commands to change for 4.3, can someone make a post with the systemd-cryptsetup commands and files?

Also it looks like they fixed the USB issue in November, so now the first post’s attach command should work by default now right?

1 Like
73

Can this be combined with Heads? Is there any documentation or gotchas other than changing the boot device to the usb (/dev/sda or sdb, etc. depending on your disk setup)? I searched this topic for ‘heads’, as well as a search engine, and didn’t find anything relevant, and I skimmed the thread, but it’s a long one so pardon me if this is elsewhere. @Insurgo ?

Also, would it be theoretically possible to have multiple machines use the same USB for the detached /boot and header?

1 Like
74

I also want to know this. It seems our first question is answered here:

But I also want to know, and I’m sorry if this is covered and I just didn’t understand it, can you have a boot partition/detached LUKS header and HEADS on the same USB, such as Nitrokey Storage? @apparatus or anyone who knows, I’d love to know, this is extremely useful for my needs and just what I needed but I am also poor.

75

Hello. I very appriciate your fix. It already helped me install Qubes 4.3 once. But you really have to edit this comment and add the hostonly=no line. Because this may confuse people who use your method. Like this already confused me today. I re-installed Qubes 4.3 again, watched your fixes in this comment but forgot that you added hostonly=no line later and installed Qubes without it, so have broken installation again. If this missing line were present in this comment this wouldn’t happen.

76

Thank you for producing this guide. I took the liberty of organising the instructions and included relevant fixes and adjustments, added some additional notes, and formatted the instructions for smoother execution and clarity. The instructions are to be found both here and in the following git repository:

Assumptions

For this guide, the devices are assumed to be:

Purpose Device
Qubes OS installer USB drive /dev/sda
USB bootloader and detached LUKS header drive /dev/sdb
Internal disk /dev/nvme0n1
Qubes OS installation partition /dev/nvme0n1p1
EFI System Partition on USB bootloader drive /dev/sdb1
/boot partition on USB bootloader drive /dev/sdb2
Detached LUKS header partition on USB bootloader drive /dev/sdb3
Opened LUKS mapper name luks
Btrfs filesystem label qubes_dom0

Warning: Device names such as /dev/sda and /dev/sdb can change between boots. Verify every device before running destructive commands.

1. Connect the USB Drives

Before proceeding, ensure that:

  • the Qubes OS installer USB drive is connected to one USB port;
  • the USB bootloader drive is connected to another USB port.

2. Configure BIOS / UEFI Settings

Ensure that virtualisation tools are enabled in BIOS / UEFI.

Ensure that Secure Boot is disabled.

3. Boot the Qubes OS Installer

Boot from the Qubes OS installer USB drive and select:

Install Qubes OS R4.3

4. Choose Language and Keyboard

Choose your language and keyboard layout.

5. Configure Time & Date

In the Time & Date menu:

  1. Set your region and city.
  2. Ensure the clock is set to the current UTC time.
  3. Click Done.

6. Configure Software Selection

Enter Software Selection.

Ensure all items are selected, then click Done.

7. Create the User

Enter User Creation.

Set the username to your desired name and configure a unique password.

Click Done.

Note: Leave the root user disabled by default.

8. Open the tty2 Shell

Drop into the tty2 shell:

Ctrl+Alt+F2

9. Verify Free Space on the Internal Disk

Verify the location and size of the unallocated free space on /dev/nvme0n1.

Note the start sector, end sector, and total number of sectors.

sfdisk --list-free /dev/nvme0n1

10. Create the Qubes OS Installation Partition

Create the Qubes OS installation partition on the internal disk:

gdisk /dev/nvme0n1

For this guide, the resulting partition is assumed to be:

/dev/nvme0n1p1

11. Overwrite the Target Devices with Random Data

Prepare the devices and partitions used in this installation by overwriting them with random data.

dd if=/dev/urandom of=/dev/nvme0n1p1 bs=1M status=progress

dd if=/dev/urandom of=/dev/sdb bs=1M status=progress

Installation location assumptions:

  • /dev/nvme0n1p1 is the Qubes OS installation partition.
  • /dev/sda is the Qubes OS installer USB drive.
  • /dev/sdb is the USB bootloader and detached LUKS header drive.

12. Create the EFI, Boot, and Header Partitions on the USB Bootloader Drive

Note: If you need to clear the partition table from the USB bootloader drive first, use:

sfdisk --delete /dev/sdb

Open the USB bootloader drive in gdisk:

gdisk /dev/sdb

Create the EFI System Partition:

n
1
2048
1230845
EF00

This creates:

/dev/sdb1

Expected size:

600 MB

Create the /boot partition:

n
2
1230848
5324799
8300

This creates:

/dev/sdb2

Expected size:

2000 MB

Note: 2 GB is used to allow room for future kernel installs during upgrades.

Create the detached LUKS header partition:

n
3
5324800
5365759
8300

This creates:

/dev/sdb3

Expected size:

20 MB

Write the changes:

w
Y

13. Create a Placeholder File for the Detached LUKS Header

Create a placeholder LUKS header file:

dd if=/dev/zero of=header.img bs=16M count=1

14. Prepare the Internal Qubes OS Partition with LUKS

Format /dev/nvme0n1p1 as a LUKS volume using the detached header file:

cryptsetup -v \
  -h sha512 \
  -c aes-xts-plain64 \
  -s 512 \
  -i 10000 \
  -y \
  --force-password \
  --header header.img \
  luksFormat /dev/nvme0n1p1

Open the LUKS volume:

cryptsetup luksOpen \
  --header header.img \
  /dev/nvme0n1p1 \
  luks

The opened mapper device should now be:

/dev/mapper/luks

15. Create the Btrfs Filesystem for Qubes OS

Create the Btrfs filesystem inside the opened LUKS volume:

mkfs.btrfs \
  --csum xxhash \
  -L qubes_dom0 \
  -d single \
  /dev/mapper/luks

16. Configure the Disks from the GUI Installer

Return to the GUI installer:

Ctrl+Alt+F6

Then:

  1. Enter System > Installation Destination.
  2. Without selecting any disks, click Refresh... at the bottom right.
  3. Click Rescan Disks.
  4. Click OK when finished.
  5. Select both the internal disk and the USB bootloader drive.
  6. Select Advanced Custom (Blivet-GUI).
  7. Enter the menu by clicking Done at the top left.

17. Configure the EFI System Partition in Blivet-GUI

From within the BLIVET GUI PARTITIONING screen:

  1. Select the USB bootloader drive, usually /dev/sdb.
  2. Select the first partition created, /dev/sdb1, which should be around 600 MB.
  3. Click the button with the illustration of cogs.
  4. Set the filesystem formatting to EFI System.
  5. Set the mount point to:
/boot/efi
  1. Click OK.

18. Configure the /boot Partition in Blivet-GUI

From within the same USB bootloader drive:

  1. Select the second partition created, /dev/sdb2, which should be around 2000 MB.
  2. Click the button with the illustration of cogs.
  3. Set the filesystem formatting to ext4.
  4. Set the mount point to:
/boot
  1. Click OK.

19. Configure the Qubes OS Root Filesystem in Blivet-GUI

Switch the primary selection to the Btrfs volume listed as:

qubes_dom0

Click the + sign button.

In the pop-up menu:

  1. Set the name to:
root
  1. Set the mount point to:
/
  1. Click OK.

20. Begin the Installation

Click Done.

Then click Accept changes.

Verify that the rest of the installer is configured as desired.

Click Begin Installation.

Wait for the installation to complete.

Important: After the installation completes, do not reboot yet.

Return to the tty2 shell:

Ctrl+Alt+F2

21. Chroot into the Installed System and Create Swap

Copy the detached LUKS header file into the installed system:

cp header.img /mnt/sysroot/root/

Enter the installed system:

cd /mnt/sysroot
chroot /mnt/sysroot/

Create the swap subvolume:

btrfs subvolume create /swap

Create the swap file:

btrfs filesystem mkswapfile \
  --size=4g \
  --uuid clear \
  /swap/swapfile

22. Write the Detached LUKS Header to the USB Bootloader Drive

Write the detached LUKS header to /dev/sdb3:

dd if=/root/header.img of=/dev/sdb3 bs=1M status=progress

23. Configure /etc/fstab

Open /etc/fstab:

nano /etc/fstab

Add the noauto option to the /boot and /boot/efi lines.

Leave the rest of the values as their defaults.

Example original /boot line:

UUID=8f4d6b2e-3a71-4c9f-b8e2-61c4d0a7f93b /boot ext4 defaults,discard 1 2

Example modified /boot line:

UUID=8f4d6b2e-3a71-4c9f-b8e2-61c4d0a7f93b /boot ext4 defaults,discard,noauto 1 2

Add the following line to the end of /etc/fstab:

/swap/swapfile none swap defaults,discard=once 0 0

Save and exit nano.

24. Configure GRUB

Open the GRUB configuration file:

nano /etc/default/grub

Modify the following lines:

GRUB_TIMEOUT=5
GRUB_DEFAULT=0
GRUB_DISABLE_SUBMENU=false

On the GRUB_CMDLINE_LINUX line:

  1. Remove the rd.luks.uuid=XXXX option.
  2. Add:
rd.driver.pre=btrfs

The line should resemble the following:

GRUB_CMDLINE_LINUX="rd.driver.pre=btrfs plymouth.ignore-serial-consoles X.X.X.qubes.fcXX.x86_64 x86_64 rhgb quiet"

Save and exit nano.

25. Configure Dracut

Create and open the Dracut crypt configuration file:

nano /etc/dracut.conf.d/crypt.conf

Add the following lines:

add_dracutmodules+=" crypt "
install_items+=" /root/header.img /usr/sbin/cryptsetup /etc/crypttab "
hostonly=no

Save and exit nano.

26. Configure /etc/crypttab

List the disk IDs for the internal NVMe disk:

ls -la /dev/disk/by-id/ | grep /nvme0n1

Search for the ID of the partition where Qubes OS was installed.

The eui ID is preferred.

The result should resemble the following:

lrwxrwxrwx 1 root root 13 March 10 12:55 nvme-eui.c3f7b29a6e115p70-part1 -> ../../nvme0n1p1

Append the relevant disk ID to /etc/crypttab.

For example, if the relevant ID ends with 5p70-part1, use:

ls /dev/disk/by-id/ | grep '5p70-part1' >> /etc/crypttab

Note: In the grep search above, specify the last few unique characters of the relevant ID and include the partition suffix, such as part1.

Open /etc/crypttab:

nano /etc/crypttab

Replace the appended raw disk ID with a complete crypttab line:

luks /dev/disk/by-id/YOUR-APPENDED-DISK-ID-HERE none header=/root/header.img,force,x-initrd.attach,discard

Example:

luks /dev/disk/by-id/nvme-eui.c3f7b29a6e115p70-part1 none header=/root/header.img,force,x-initrd.attach,discard

Save and exit nano.

27. Rebuild GRUB and Regenerate Dracut

Rebuild the GRUB configuration:

grub2-mkconfig -o /boot/grub2/grub.cfg

Regenerate the initramfs images:

dracut -f --regenerate-all

Exit the chroot:

exit

Reboot:

reboot

28. Remove the Installer USB Drive

During the reboot, the Qubes OS installer USB drive may be removed from the USB port.

Leave the USB bootloader drive connected.

29. Boot from the USB Bootloader Drive

Upon rebooting:

  1. Press Enter at the boot screen.
  2. Enter the boot selection menu.
  3. Select the USB bootloader drive.
  4. Enter the encryption key.
  5. Once fully decrypted, proceed to configure the system by selecting QUBES OS and entering the menu.

30. Complete Qubes OS Initial Configuration

In the Qubes OS initial configuration menu:

  1. Leave all default options selected.
  2. Also select Enable system and template updates over the Tor anonymity network using Whonix.
  3. Click Done.
  4. Click FINISH CONFIGURATION.

Qubes OS will now complete its installation.

31. Reboot Before First Login

Perform another fresh reboot before first logging into Qubes OS.

1 Like
77

Would any of you consider doing this guide solely on detached /boot and header? Currently, it looks like two different guides: this, along with BTRFS. I’d like to have a detached /boot and header without BTRFS, but I don’t know enough to remove BTRFS from the guide, and I don’t have the time to brute-force it.

If you don’t want to do this, could you point me in the right direction or maybe give some pointers on how to figure it out myself from the guide?

78

I could be wrong but the post linked below seems to explain how to construct the LVM structure instead of BTRFS. Also note that the last comment has updated instructions for 4.3 (just the edits)

I would assume just use this in place of the BTRFS part of the guide, but I haven’t tested or even read the post myself.

Let me know how it goes!

79

Thanks! I’m working on it now. It’s going well, but there are a few changes I’d heavily suggest from this: argon2(i/d/id) should be preferred over sha512, since it was designed for this function whereas SHA was not. Second, the EFI partition should be vFAT (vfat in the GUI), as ‘EFI Partition’ is nowhere to be found on 4.3 installer. Third, it should be noted that some installations require a 1MiB BIOSboot partition at the beginning in order to boot correctly.

I had a problem, but a reinstall fixed it.

That being said, I got all the way to the end and when I rebooted, it just looped at dracut expecting the qubes_dom0/root to be available before it prompted me for the password. Specifically, it hangs on
[ OK ] Finished systemd-vconsole-setup.service - Virtual Console Setup.
then loops something about dracut-initqueue, and then
Starting dracut-emergency.service - Dracut Emergency Shell…
Warning: /dev/mapper/qubes_dom0/root does not exist
Warning: /dev/qubes_dom0/root does not exist
Warning: /dev/qubes_dom0/swap does not exist
and then gives an emergency shell.
Do you know what may have happened?

80

Hi, I am also looking at building this to work with Heads and also need to do this without BTRFS. (ext4) Did you by chance succeed?