Qubes OS Installation - BTRFS w/ Detached encrypted boot and header

51lieal · October 26, 2021, 7:37pm

Continuing the discussion from Qubes OS Installation - Detached encrypted boot and header:

In above thread I use xfs with uefi + gpt.
In here I will using mbr + bios with btrfs blake2 checksum.

Keep in mind that below are disk i used in the tutorial, you can use 2 flashdrive (1 boot, 1 header) + 1 hdd or whatever you want.

and btw I don’t detach swap partition, well i think you should. Do write out to /dev/sdb3 if you want

Take this as additional challenge

I use different chipper for each partition; root with aes-xts, swap with serpent-xts and boot with twofish-xts, you can go with aes-xts for better performance.

/dev/sda = system
/dev/sdb = flashdrive

Don’t forget to change boot order, so flashdrive is booting first.

Please watch out any space, slash, periode in command issue / files IT REALLY MATTER

After booting into installation in language section, press ctrl + alt + f2

—# WARNING CONFIRM YOUR DISK FIRST BEFORE EXECUTING COMMAND

[anaconda /] dd if=/dev/urandom of=/dev/sda bs=1M status=progress
[anaconda /] dd if=/dev/urandom of=/dev/sdb bs=1M status=progress

Using /dev/urandom will take longer than /dev/zero but more secure.

[anaconda /] fdisk /dev/sda
—# root partition
n
p
1
(enter)
+36GiB

—# swap partition
n
p
2
(enter)
(enter) #4GiB for swap

w

Create 2 or 3 partition in usb drive.

[anaconda /] fdisk /dev/sdb
—# boot partition
n
p
1
(enter)
+1GiB

—# header partition
n
p
2
(enter)
+30MiB

—# you can create 3 partition for additional use (whatever it is) / just use 2 partition
n
p
3
(enter)
(enter)

w

My block device after partitioning :

—# I use iter time 1 for speeding up decrypt process you should increase it in real installation, see 5.13 for details.
[anaconda /] cryptsetup -c aes-xts-plain64 -h sha512 -s 512 -y -i 1 --use-random luksFormat /dev/sda1
[luks prompt /] YES
[luks prompt /] (enter password)
[luks prompt /] (verify password)
[anaconda /] cryptsetup -c serpent-xts-plain64 -h sha512 -s 512 -y -i 1 --use-random luksFormat /dev/sda2
[luks prompt /] YES
[luks prompt /] (enter password)
[luks prompt /] (verify password)
[anaconda /] cryptsetup luksOpen /dev/sda1 root
[luks prompt /] (enter password)
[anaconda /] cryptsetup luksOpen /dev/sda2 swap
[luks prompt /] (enter password)

[anaconda /] mkfs.btrfs --csum blake2b -L qubes_dom0 -d single /dev/mapper/luksroot
[anaconda /] mkswap /dev/mapper/swap

Back to gui with ctrl + alt + f6.
Choose language, timezone, user, and lastly storage.
Click refresh on bottom right and rescan disk.
Select disk sda and sdb, storage configuration is Advanced Custom (Blivet-GUI) click done.

—# /dev/sda

Right click qubes_dom0 > new > name = root, mountpoiont = /

—# /dev/sdb

Right click on sdb1 > edit > format to ext2 / ext4 and mountpoint to /boot leave name to none.

Click done, and this is the Summary of Changes

Click done and begin installation.
After completion, switch back to shell with ctrl + alt + f2

My block device after installing :

[anaconda /] chroot /mnt/sysroot/
[anaconda /] mount -oremount,ro /boot
[anaconda /] install -m0600 /dev/null /tmp/boot.tar
[anaconda /] tar -C /boot --acls --xattrs --one-file-system -cf /tmp/boot.tar .
[anaconda /] umount /boot
—# WARNING CONFIRM YOUR DISK FIRST BEFORE EXECUTING COMMAND
[anaconda /] dd if=/dev/urandom of=/dev/sdb1 bs=1M
[anaconda /] cryptsetup -c twofish-xts-plain64 -h sha512 -s 512 -y -i 1 --use-random --type luks1 luksFormat /dev/sdb1
[luks prompt /] YES
[luks prompt /] (enter password)
[luks prompt /] (verify password)
—#
[anaconda /] uuidR=“$(blkid -o value -s UUID /dev/sda1)” # root device
[anaconda /] uuidB=“$(blkid -o value -s UUID /dev/sdb1)” # boot device
[anaconda /] uuidS=“$(blkid -o value -s UUID /dev/sda2)” # swap device
[anaconda /] cryptsetup luksOpen /dev/sdb1 luks-$uuidB
[anaconda /] mkfs.ext2 -m0 -U $uuidB /dev/mapper/luks-$uuidB

—# Configure fstab

Change UUID=…on boot and root line to /dev/mapper/luks-(your $uuidR and $uuidB)
Add swap and leave the rest to default value

[anaconda /] mount -v /boot
[anaconda /] tar -C /boot --acls --xattrs -xf /tmp/boot.tar

—# Configure grub
[anaconda /] echo “GRUB_ENABLE_CRYPTODISK=y” >> /etc/default/grub

in GRUB_CMDLINE_LINUX delete all of rd.luks… then add cryptdevice=$uuidB:luks-$uuidB in my case this is my final grub_cmdline_linux line :

GRUB_CMDLINE_LINUX="cryptdevice=aa7332a5-e4ac-442c-9328-cdbd4a0b42e8:luks-aa7332a5-e4ac-442c-9328-cdbd4a0b42e8 plymouth.ignore-serial-consoles i915.alpha_support=1 rd.driver.pre=btrfs rhgb quiet"

—# create luks keys so we dont have to enter any password after grub
[anaconda /] mkdir -m0700 /etc/keys
[anaconda /] ( umask 0077 && dd if=/dev/urandom bs=1 count=64 of=/etc/keys/root.key conv=excl,fsync )
[anaconda /] ( umask 0077 && dd if=/dev/urandom bs=1 count=64 of=/etc/keys/boot.key conv=excl,fsync )
[anaconda /] ( umask 0077 && dd if=/dev/urandom bs=1 count=64 of=/etc/keys/swap.key conv=excl,fsync )
[anaconda /] cryptsetup luksAddKey /dev/sda1 /etc/keys/root.key
[luks prompt /] (system password)
[anaconda /] cryptsetup luksAddKey /dev/sda2 /etc/keys/swap.key
[luks prompt /] (swap password)
[anaconda /] cryptsetup luksAddKey /dev/sdb1 /etc/keys/boot.key
[luks prompt /] (boot password)
[anaconda /] cryptsetup luksHeaderBackup /dev/sda1 --header-backup-file header
—# WARNING CONFIRM YOUR DISK FIRST BEFORE EXECUTING COMMAND
[anaconda /] dd if=/header of=/dev/sdb2 bs=16M count=1 status=progress
[anaconda /] shred -uvz /header
[anaconda /] shred -uvz /tmp/boot.tar

—# Configure crypttab
[anaconda /] echo -e “luks-$uuidR /dev/sda1 /etc/keys/root.key luks,discard,key-slot=1,header=/dev/sdb2 \nluks-$uuidS UUID=$uuidS /etc/keys/swap.key luks,key-slot=1 \nluks-$uuidB UUID=$uuidB /etc/keys/boot.key luks,key-slot=1” > /etc/crypttab

—# Configure dracut
[anaconda /] echo -e “add_dracutmodules+=" crypt " \ninstall_items+=" /etc/keys/*.key "" > /etc/dracut.conf.d/misc.conf

[anaconda /] vi /usr/lib/dracut/modules.d/90crypt/module-setup.sh

—# write a persistence device at /etc/block_uuid.map in generated initramfs
echo “/dev/sda1 $uuidR
/dev/disk/by-uuid/$uuidB $uuidB
/dev/disk/by-uuid/$uuidS $uuidS” > “${initdir}/etc/block_uuid.map”

—# write a persistence device at /etc/crypttab in generated initramfs (have try inject /etc/crypttab into initramfs but it doesn’t match, so we’ll rewrite again)

echo “luks-$uuidR /dev/sda1 /etc/keys/root.key luks,discard,key-slot=1,header=/dev/sdb2
luks-$uuidB UUID=$uuidB /etc/keys/boot.key luks,key-slot=1
luks-$uuidS UUID=$uuidS /etc/keys/swap.key luks,key-slot=1” > $initdir/etc/crypttab

[anaconda /] grub2-install --recheck /dev/sdb
[anaconda /] grub2-mkconfig -o /boot/grub2/grub.cfg
[anaconda /] dracut -v -f /boot/initramfs-*
[anaconda /] exit
[anaconda /] umount /mnt/sysroot/boot
[anaconda /] umount -l /mnt/sysroot
[anaconda /] umount -l /mnt/sysimage
[anaconda /] swapoff /dev/mapper/luksswap
[anaconda /] cryptsetup luksClose /dev/mapper/luksroot
[anaconda /] cryptsetup luksClose /dev/mapper/luks-$uuidB
[anaconda /] cryptsetup luksClose /dev/mapper/luksswap
[anaconda /] cryptsetup luksErase /dev/sda1
[luks prompt /] YES
[anaconda /] wipefs -a /dev/sda1

My block device after configure everything :

[anaconda /] reboot

—# Screenshot

After update, everything still works.

ewokky · October 27, 2021, 5:11am

Send me your btc wallet address and I am buying you dinner for this. Thank you.

ymy · October 27, 2021, 11:22am

@adw i couldnt find this as enhancement issue, but it sounds like a great feature, other OS (SuSe Tumbleweed, Endeavour OS [arch distro]) have this a s default. probably anti-evil-maid needs to be adjusted. Should i open one with link to these posts or would that just spam the issue tracker?

adw · October 27, 2021, 7:11pm

I honestly don’t know how feasible it would be to integrate this into the installer. It seems highly likely that it would be a help wanted issue. TBH, it seems like one of those issues that’s likely to sit in the tracker for years without any volunteers, but you never know. Maybe someday. Bearing in mind those caveats, feel free to open an enhancement issue, if you like!

ymy · October 27, 2021, 7:46pm

i am more of an avid user and not able to put any bounty on anything.
thank you for your estimation and as long as there is no majority who would think it could contribute security i’ll pass on opening new stale issues

51lieal · October 27, 2021, 9:03pm

What do you mean by those os has this feature as default? you mean they did detach header when installing ? or just encrypt everything including boot ?

ymy · October 27, 2021, 9:31pm

yeah the encrypted boot partition. sorry for the mixup.

jodobear · December 11, 2021, 11:27pm

System:

EFI
single nvme disk
no other OS

I have followed this bible to this point, but this pass seems unsurmountable! I’m getting:

grub2-install: error: /usr/lib/grub/x86_64-efi/modinfo.sh doesn't exit. Please specify --target --directory

Grub only has i386! That’s strange. It should at least have x86_64 even if not efi

$ ls /usr/lib/grub/
i386-pc

I tried grub-install --target=x86_64-efi & --efi-directory=/boot/efi --bootloader-id=grub2 which returned the same error as /boot/efi isn’t mounted. I looked into the ESP partition, but it doesn’t have the file.

So close and yet so far! I really would like to complete this, it’d feel like a major achievement. Thank you @51lieal for this beauty!

This is highly underrated requirement. Why is this not default? Because, you’d want to encrypt everything, even the boot. Why leave it unencrypted? And with the header detached it’s even better.

EDIT:

Interpreting your comment

I continued without grub configuration

I finished the install but, i get the following error and then goes back to Grub:

error: no such evice: some-UUID.
Loading Xen 4.14.3 ...
error: file '/xen-4.14.3.gz' not found.
Loading Linux 5.10.76-1.fc32.qubes.x86_64 ...
error: you need to load the kernel first.
Loading initial ramdisk ...
error: you need to load the kernel first.

Press any key to conttinue...
error: file /grub2/themes/qubess/theme.txt' not found.

Press any key to conttinue...

On going to Rescue QubesOS on the installation drive, after entering the passphrase for /boot & /header i.e. /root (i assume) i get the message You don't have any Linux partitions. Reebooting

Why do you erase the /root partition?

51lieal · December 12, 2021, 1:11am

The problem is this guide is for

you should read

for uefi.

if you read uefi guide, i did [anaconda /] cp -r /usr/lib/grub/x86_64-efi /mnt/sysroot/boot/efi/EFI/qubes/

it’s not erasing root partition but erasing all keyslots and make the LUKS container permanently inaccessible.

then

wipefs can erase filesystem, raid or partition table signatures (magic\ strings) from the specified device to make the signature invisible for\ libblkid

i did this because we doing detachable header. so the partition is remain unknown.

If you want system that use btrfs, simply do custom storage

delete all partition, then change lvm thin provisioning to btrfs and “click here to create them automatically” and you done.

jodobear · December 12, 2021, 2:23am

But, doing the following wouldn’t create an encrypted /boot, right? For that i’d have to follow your guide, right?

If i use the BTRFS parts from this guide and the EFI ones from Qubes OS Installation - Detached encrypted boot and header, that should work?

Could you kindly point me to where can i learn/read more about this?

Thank you!

51lieal · December 12, 2021, 10:56am

Indeed, my guide is not explaining anything, it’ just “how to”, because i believe that many people are already understanding what is detach header and encrypted boot.

yes combine both guide for reference.

This guide is really help me a lot, take a look Full disk encryption, including /boot: Unlocking LUKS devices from GRUB

And i did my research on

Because we doing detached header.

If you not doing detach header, following what debian cryptsetup team write is enough.

orangic · February 8, 2022, 4:02am

Hello, I followed your guide step-by-step except I put swap partition to the flashdrive also. However, after reboot and entering my boot partition password in grub, Qubes asked for my luks passwords for both the root and boot partitions several times, and eventually failed to enter the system. My screen was below. “Extreme” is my usb flash drive.

My laptop is a Librem 15v3. I have no idea what I did wrong. Could you help?
Anyway, thank you for the great guide!

51lieal · February 9, 2022, 10:59am

based on that, i assume you not using luks keys too decrypt drive right ?

Are you using uefi / legacy bios ?

orangic · February 9, 2022, 1:01pm

I’m pretty sure I did add each of the three keys to its corresponding luks partition. Because my root partition is on the SSD of my laptop, and swap on usb drive sdb3 and boot on sdb1, what I actually did was:

[anaconda /] cryptsetup luksAddKey /dev/nvme0n1p1 /etc/keys/root.key
[luks prompt /] (system password)
[anaconda /] cryptsetup luksAddKey /dev/sdb3 /etc/keys/swap.key
[luks prompt /] (swap password)
[anaconda /] cryptsetup luksAddKey /dev/sdb1 /etc/keys/boot.key
[luks prompt /] (boot password)

I installed 3 times and still got the same problem.

It’s legacy. The laptop does not support uefi.

51lieal · February 9, 2022, 5:42pm

I’m sure that the problem is either in your

/etc/dracut.conf.d/misc.conf
/usr/lib/dracut/modules.d/90crypt/module-setup.sh

can you boot into qubes os rescue disk, skip to shell,
open your root partition, mount it to mnt and look for those files.

please ss at the place where i ss in the guide above (module-setup.sh)

kafix · November 25, 2022, 1:55pm

rm

kafix · November 25, 2022, 2:03pm

First of all, I would like to appreciate and thank you for the effort you put into writing this how-to. I think it can be very useful for many Q-OS users! I just want to add a few minor additions.

Firstly (and most importantly) I think that modifying the dracut script is not the best idea: with a possible update, the file may be replaced with a new version and we will lose the configuration entries.
I added the key files to the /etc/cryptsetup-keys.d/ directory I created myself and the appropriate lines to the /etc/crypttab file:

<map_name> <UUID=…> /etc/cryptsetup-keys.d/<key_file_name> luks,discard

(discard option due to the use of SSDs)

This does not require any additional modifications to the kernel commandline or manual adding definition of: /etc/cryptsetup-keys.d/* to dracut configuration anywhere.

Secondly, I don’t use the key-slot=1 option to also be able to enter the password manually (placed in some other slot). This can be useful when changing something later in the configuration you accidentally make a mistake related to the definition, location or other changes to the key files.

Thirdly, for my setup, I also don’t use mapping names like luks-XXXX… but simple, plain names like ‘boot’, ‘root’ or ‘swap’. I don’t know if this is against some general convention used in Fedora (I use various distros mostly on ZFS not using systemd), but configuration is more human readable. In addition, the lsblk, mount or df commands show device names in the shorter simple form /dev/mapper/<name> (I also use the same format in the /etc/fstab file).

Everything works fine (I enter the password only once for grub, at system startup, other partitions are decrypted automatically by the system).

I’m using Q-OS 4.1 on UEFI but I think the same should work in LEGACY.