Install Qubes OS with boot partition and a detached LUKS header on USB

[wilsonb]

What is the benefit of a detached LUKS header?

[apparatus]

I’ve added a note about it in the guide:

@apparatus

The purpose of having a detached LUKS header and boot partition on a separate external disk is to achieve a deniable encryption:
Deniable encryption - Wikipedia
The encrypted disk will look like an unused/empty unpartitioned disk.

[robertford]

@LadyPropane

But I feel like Anaconda may be a bit unpredictable

I have had similar experiences and would like to confirm that Anaconda does unexpected things if you don’t follow @apparatus guide step-by-step.

[wilsonb]

Thanks for the additional info!

1. So the benefit is deniability, since only the header shows that the encrypted data is “real”. Without it, it just looks like randomly overwritten data. Is that correct?

2. Is there a security benefit?

3. What are the risks? I would be concerned about destroying my data in any way. Is this a valid concern?

Thank you very much!

[apparatus]

@wilsonb

So the benefit is deniability, since only the header shows that the encrypted data is “real”. Without it, it just looks like randomly overwritten data. Is that correct?

Yes.

@wilsonb

Is there a security benefit?

For detached header - no.
For boot partition on a separate external drive - if you multiboot in other OSes on this computer then if you remove the drive with Qubes OS boot partition then they won’t be able to modify the files on boot partition.

@wilsonb

What are the risks?

Anti evil maid (AEM)

@wilsonb

I would be concerned about destroying my data in any way. Is this a valid concern?

If you lose the USB drive with your detached header and you didn’t make a backup of it then you’ll lose the data.

[wilsonb]

As always, thank you for your time and information!

@apparatus

For boot partition on a separate external drive - if you multiboot in other OSes on this computer then if you remove the drive with Qubes OS boot partition then they won’t be able to modify the files on boot partition.

That’s a good point! It would also allow me to avoid unwanted physical tampering. But the question is, if I have to worry about such advanced attacks, is it wise to only worry about the software part? I would say that with such a high threat model, you should glue your entire PC to your body, not just the boot partition. Well, I guess every step can help.

@apparatus

If you lose the USB drive with your detached header and you didn’t make a backup of it then you’ll lose the data.

That can be an advantage as well, I guess.

Are there any other risks? Do I need to update the LUKS header? I could see a day when LUKS might update some internal crypto at the moment I unlock the drive. If I am not constantly up to date with LUKS and Linux encryption, I might miss that moment and then unknowingly invalidate my backups. Is this a valid concern? Anything else?

So if I understand the benefit correctly, it is all about deniability. But how important is that? Don’t get me wrong, I’m all for security and privacy. But the two main scenarios I see are:

1. Some legal entity sees my encrypted data (with or without the LUKS header) but doesn’t do anything because they are bound by law. In this case, it might make more sense to just have the LUKS header like any other Linux user. This would avoid any additional unwanted suspicion.

2. Some entity with a $5 wrench attacks me because it’s pretty obvious that I’m not carrying around a laptop with useless random data.

I would say that in a legal environment (not the guys hitting me over the head with a wrench) I could claim that I do not have the LUKS header and therefore cannot unlock my drive.

Three follow-up questions:

1. Is it true that without the LUKS header there is no way to prove that the password I give out is the right one?

2. Is it true that without the LUKS header, even if I give out the correct password (or it leaks out), no one can decrypt the disk because essential crypto parameters (keys) are stored on that header? I am a crypto novice, but I would assume that my password decrypts the encryption key stored on the LUKS header, which then decrypts the drive. Is this correct?

3. What would happen if my LUKS header was leaked? It might make sense to upload it to Google Drive (or somewhere else) and not travel with it. That way you could literally make your data inaccessible as you don’t have the keys. Is that wise? Is the LUKS header itself protected by my password? What data would be exposed?

[apparatus]

@wilsonb

Do I need to update the LUKS header? I could see a day when LUKS might update some internal crypto at the moment I unlock the drive. If I am not constantly up to date with LUKS and Linux encryption, I might miss that moment and then unknowingly invalidate my backups. Is this a valid concern?

You only need to update the LUKS header when you plan to make some change to the encryption algorythm, add new keys etc. It’s not changed with something like system update.
The detached header is a standard LUKS feature so using shouldn’t break anything.

@wilsonb

Some legal entity sees my encrypted data (with or without the LUKS header) but doesn’t do anything because they are bound by law. In this case, it might make more sense to just have the LUKS header like any other Linux user. This would avoid any additional unwanted suspicion.

There are laws in some countries (e.g. UK) that force you to give up the encryption keys to the law enforcement or go to jail.
But if it’s impossible to prove that your disk contains any encrypted data then you shouldn’t be prosecuted by law.
Just have a second drive with a dummy OS there for a cover so you can that the second drive is just unused and it won’t look too suspicious.

@wilsonb

I would say that in a legal environment (not the guys hitting me over the head with a wrench) I could claim that I do not have the LUKS header and therefore cannot unlock my drive.

It doesn’t work like this in some countries.

@wilsonb

Is it true that without the LUKS header there is no way to prove that the password I give out is the right one?

Yes.

@wilsonb

Is it true that without the LUKS header, even if I give out the correct password (or it leaks out), no one can decrypt the disk because essential crypto parameters (keys) are stored on that header? I am a crypto novice, but I would assume that my password decrypts the encryption key stored on the LUKS header, which then decrypts the drive. Is this correct?

Yes.

@wilsonb

What would happen if my LUKS header was leaked? It might make sense to upload it to Google Drive (or somewhere else) and not travel with it. That way you could literally make your data inaccessible as you don’t have the keys. Is that wise? Is the LUKS header itself protected by my password? What data would be exposed?

If you have some easy password that could be bruteforced or there is a vulnerability in the encryption algorithm then the decryption key could be compromised.
You can encrypt the header additionally before uploading it somewhere.
You can store the header on the microSD card and hide it somewhere in your baggage when you travel.
Or you can send the microSD containing the additionally encrypted header using mail to the destination, travel there without it and get it when you arrive safely.

[LadyPropane]

TL;DR I copied my hard drive to a new one with dd and then had to update the disk ID in the initramfs in order to boot. To do that, I booted Qubes from the old hard drive, mounted /boot, changed /etc/crypttab and ran the grub2-mkconfig and dracut commands from the tutorial. Afterwards I booted from the new drive with no issues.

Longer version (posted in case it might be helpful to anyone):
After successfully following this tutorial, I wanted to switch my HDD for a faster SSD. The new SSD was the exact same number of bytes as the HDD so I copied the HDD to the SSD with dd, but afterwards I couldn’t boot since the disk ID of the new SSD was different than the disk ID of the old HDD.

When I tried to boot with the new SSD, I got a black screen for a few minutes, and then repeating errors referencing the UUID of the decrypted disk (dm-0 in /dev/disk/by-uuid):

[ 228.682835] dracut-initqueue[448]: Warning: dracut-initqueue: timeout, still waiting for following initqueue hooks:
[ 228.689096] dracut-initqueue[448]: Warning: /bit/dracut/hooks/initqueue/finished/devexists-\x2fdev\x2fdisk\x2fby-uuid\x2f<UUID>.sh: "if !
grep -q After=remote-fs-pre.target /run/systemd/generator/systemd-cryptsetup@*.service 2>/dev/null; then
[ 228.689197] dracut-initqueue[448]:     [ -e "/dev/disk/by-uuid/<UUID>" ]
[ 228.689244] dracut-initqueue[448]: fi"
[ 228.694295] dracut-initqueue[448]: Warning: dracut-initqueue: starting timeout scripts
[ 228.695118] dracut-initqueue[448]: Warning: Could not boot.

But since the UUID couldn’t have changed when copying the HDD to the SSD, I figured out that the only thing that changed was the disk ID.

I had to regenerate the initramfs which contains the disk ID. Here are the steps I took.

I booted with the old HDD. I kept the USB drive with the detached /boot and LUKS header plugged in.
I ran all the commands in dom0 as root (sudo su). Probably not the best practice, but I wasn’t going to use that HDD anymore anyway, so I didn’t care much about following best practices on that install.
I mounted /boot and /boot/efi. Not sure if I only needed to mount /boot for this to work.

dom0# qvm-block a dom0 sys-usb:sda1
dom0# qvm-block a dom0 sys-usb:sda2
dom0# mount /boot
dom0# mount /boot/efi

(mount knows which partition to mount as /boot and which partition to mount as /boot/efi because of the entries in /etc/fstab we made during the installation).

I attached the new SSD via an USB dock, attached it to dom0 (again, not a best practice).

dom0# qvm-block a dom0 sys-usb:sdb
dom0# ls -l /dev/disk/by-id

After I saw the new disk ID (ata-...), I updated /etc/crypttab with it:

dom0# vi /etc/crypttab

and then ran these commands:

dom0# grub2-mkconfig -o /boot/grub2/grub.cfg
dom0# dracut -f --regenerate-all

I’m not sure if grub2-mkconfig was needed at all - probably not.

Afterwards I swapped the HDD with the new SSD in the laptop and booted with no issues on the SSD. Of course, /etc/crypttab in dom0 still had the disk ID of the old HDD because I copied the HDD’s contents to the SSD before changing /etc/crypttab on the HDD. I changed it to use the new ID, but I don’t think it matters because we use /etc/crypttab from the initramfs when booting, not from the root filesystem after we decrypt it.

As a precaution, I made a backup of the whole USB drive on another computer before running anything.

I also wanted to look at the initramfs contents before and after doing this, just to make sure it changed with the new ID.
Before I regenerated the initramfs with dracut:
I copied the initramfs to a temporary directory:

dom0# mkdir /tmp/i
dom0# cp /boot/initramfs-6.6.48-1.qubes.fc37.x84_64.img /tmp/i
dom0# cd /tmp/i

I extracted the initramfs to check where the disk ID was being stored (it should be in etc/crypttab). I followed this guide for extracting the initramfs:

dom0# cpio -ivF initramfs-6.6.48-1.qubes.fc37.x84_64.img

The output showed “30 blocks”, so I had to skip 30 blocks with dd:

dom0# dd if=initramfs-6.6.48-1.qubes.fc37.x84_64.img skip=30 of=main_archive
dom0# mkdir initramfs_dir
dom0# zcat main_archive | cpio -ivD initramfs_dir
dom0# cd initramfs_dir

I searched the disk ID of the HDD inside the extracted initramfs.

dom0# grep -rli <part_of_disk_ID> *

The output was:

etc/block_uuid.map
etc/crypttab

etc/crypttab was expected. etc/block_uuid.map contained only /dev/disk/by-id/<disk_ID>.
After I regenerated the initramfs with dracut:
I followed the same process - copied /boot/initramfs* to /tmp/i2, extracted it and saw that the 2 files now contain the disk ID of the new SSD. I forgot to check if anything else in the initramfs or on the USB drive as a whole had changed.

I posted this because messing with initramfs isn’t something I do regularly, so at first I was a bit apprehensive and confused about some things and it took me a while to run a few simple commands.

Finally, I’m wondering what the correct steps would be if I switched from a smaller drive to a bigger drive, or the other way around. Just guessing, but if we were switching to a smaller drive, we could first shrink the btrfs sumvolume, then the btrfs filesystem, then the LUKS partition somehow. Or, if we were switching to a bigger drive, we would expand the LUKS partition, then the btrfs filesystem, then the btrfs sumvolume? I’m not sure which of those operations are possible or safe. Everything would likely need to be done after booting from a UBS drive with a live Linux install, after we copy header.img to it from the initramfs. I think I would just back up my qubes and my changes to dom0 and perform a fresh install, instead. I consider myself lucky that both the old HDD and the new SSD were exactly the same size.

[apparatus]

@LadyPropane

Just guessing, but if we were switching to a smaller drive, we could first shrink the btrfs sumvolume, then the btrfs filesystem, then the LUKS partition somehow. Or, if we were switching to a bigger drive, we would expand the LUKS partition, then the btrfs filesystem, then the btrfs sumvolume?

Yes.

@LadyPropane

I’m not sure which of those operations are possible or safe.

They are possible and should be safe.

@LadyPropane

Everything would likely need to be done after booting from a UBS drive with a live Linux install, after we copy header.img to it from the initramfs.

In case of copying from smaller to bigger disk you can do everything inside the old system.

[wilsonb]

I had other things to do. Anyway, I would appreciate it if we could continue this conversation.

@apparatus

The detached header is a standard LUKS feature so using shouldn’t break anything.

To be very clear here. You are saying that a detached LUKS header is a direct feature/design choice? Why can’t I just run it from the LUKS command or something?

@apparatus

There are laws in some countries (e.g. UK) that force you to give up the encryption keys to the law enforcement or go to jail.

I am aware of this. Do you have a list of countries where such laws exist and are enforced? I think the safest way is to assume such things for all countries since laws can change and there is also the special terrorist card the feds like to pull out on anyone they don’t like.

@apparatus

Just have a second drive with a dummy OS there for a cover so you can that the second drive is just unused and it won’t look too suspicious.

When a disk is encrypted, the disk is full of random noise (the encrypted data). Is this different from an empty disk? I assume not, otherwise plausible deniability would not work, but I am not a forensic expert. Is a disk always noisy, and empty is just the imaginary state that distinguishes the noise you intentionally created (encrypted data) from the noise of the default/empty disk state?

@apparatus

Just have a second drive with a dummy OS there for a cover so you can that the second drive is just unused and it won’t look too suspicious.

You mean a second disk with something bootable, just for plausibility? I think that is a good idea, while the argument that you swapped the disk to a new one and did not have time to install a new OS should be legally sufficient.

@apparatus

It doesn’t work like this in some countries.

Why? I could say that I am a privacy lover and an idiot at the same time. I lost the encrypted LUKS header last week. Sorry about that… Of course that is very suspicious.

@apparatus

Yes.

So LUKS works by using my password to decrypt the encryption key, which then actually decrypts my disk?

@apparatus

If you have some easy password that could be bruteforced or there is a vulnerability in the encryption algorithm then the decryption key could be compromised.
You can encrypt the header additionally before uploading it somewhere.
You can store the header on the microSD card and hide it somewhere in your baggage when you travel.
Or you can send the microSD containing the additionally encrypted header using mail to the destination, travel there without it and get it when you arrive safely.

Thanks for the clarification. So the header itself is encrypted/protected by my password? Is the entire header protected or could it be tampered with? I am asking to fully understand when it would make sense to encrypt the header itself. Obviously if I want to hide the fact that my uploaded/mailed data is a LUKS header, but what else?

[wilsonb]

I would also like to ask/clarify about the tamper protection of a LUKS header setup itself. Since nothing remains on disk that is not encrypted, simple software tampering should not be possible. Is this a correct assumption? Like the bootloader, etc.?

The firmware can still be tampered with, of course, just as the hardware can be replaced or modified.

HEADS would also measure the firmware, so a LUKS header is not as good. But it should be possible to combine LUKS header + HEADS, right?

[apparatus]

@wilsonb

You are saying that a detached LUKS header is a direct feature/design choice?

Yes, detached header is a LUKS feature, not some hack.
You can check the LUKS docs:
luks2_doc_wip.pdf · main · cryptsetup / LUKS2-docs · GitLab

@wilsonb

Why can’t I just run it from the LUKS command or something?

I didn’t understand your question, explain in more detail.

@wilsonb

Do you have a list of countries where such laws exist and are enforced?

No.
Also in some countries you could be forced to supply the password when crossing the border or you’ll be denied entry or the device will be confiscated (e.g. US).

@wilsonb

When a disk is encrypted, the disk is full of random noise (the encrypted data). Is this different from an empty disk? I assume not, otherwise plausible deniability would not work, but I am not a forensic expert. Is a disk always noisy, and empty is just the imaginary state that distinguishes the noise you intentionally created (encrypted data) from the noise of the default/empty disk state?

Secure erase will always overwrite the disk with a random data so nobody will be able to tell if this random data on the disk is a result of a secure erase or if this disk is encrypted.

@wilsonb

You mean a second disk with something bootable, just for plausibility? I think that is a good idea, while the argument that you swapped the disk to a new one and did not have time to install a new OS should be legally sufficient.

Yes.

@wilsonb

Why? I could say that I am a privacy lover and an idiot at the same time. I lost the encrypted LUKS header last week. Sorry about that… Of course that is very suspicious.

I don’t know the legal details, but if saying something like “I forgot the password” worked then the law wouldn’t work at all:
Can I be charged for not giving my password to the police? | THB Solicitors
But I guess they need to at least prove that you have the encrypted data.

@wilsonb

So LUKS works by using my password to decrypt the encryption key, which then actually decrypts my disk?

Yes.

@wilsonb

So the header itself is encrypted/protected by my password?

The encryption key stored in the header is encrypted, not the whole header itself.

@wilsonb

Is the entire header protected or could it be tampered with?

It could be tampered with. But it’d just corrupt the header without your password.

@wilsonb

I am asking to fully understand when it would make sense to encrypt the header itself. Obviously if I want to hide the fact that my uploaded/mailed data is a LUKS header, but what else?

I don’t know of other uses.

@wilsonb

I would also like to ask/clarify about the tamper protection of a LUKS header setup itself. Since nothing remains on disk that is not encrypted, simple software tampering should not be possible. Is this a correct assumption? Like the bootloader, etc.?

What do you mean by “software tampering”? The disk is still writable so you can still overwrite the data of it. The data on the disk will be destroyed so could it be considered as “software tampering”?

@wilsonb

HEADS would also measure the firmware, so a LUKS header is not as good. But it should be possible to combine LUKS header + HEADS, right?

Yes.

[wilsonb]

@apparatus

I didn’t understand your question, explain in more detail.

I would love to see a script to do a detached header setup out of the box. But I may be asking for too much.

@apparatus

Secure erase will always overwrite the disk with a random data so nobody will be able to tell if this random data on the disk is a result of a secure erase or if this disk is encrypted.

Thanks for the clarification. So there are forensic differences between empty and random noise? Not that it matters after your explanation, but I would like to know.

@apparatus

What do you mean by “software tampering”? The disk is still writable so you can still overwrite the data of it. The data on the disk will be destroyed so could it be considered as “software tampering”?

I thought about tampering with the boot loader. Of course, the disk could be overwritten or physically destroyed. My idea here is that the encrypted disk without any unencrypted data cannot be attacked by Evil Maid. Except hardware + firmware.

Thank you for all your answers!

[apparatus]

@wilsonb

I would love to see a script to do a detached header setup out of the box. But I may be asking for too much.

You mean to be able to set up detached header in Qubes OS installer GUI? The installer GUI is ANACONDA from Fedora and it has only basic functionality anything more advanced will require you to go to the console.

@wilsonb

Thanks for the clarification. So there are forensic differences between empty and random noise? Not that it matters after your explanation, but I would like to know.

Well, truly empty would be a disk filled with 0x00 or 0xFF, but with random data you can’t tell if it’s an encrypted data or just random noise.

@wilsonb

My idea here is that the encrypted disk without any unencrypted data cannot be attacked by Evil Maid. Except hardware + firmware.

Yes.

[wilsonb]

Thanks!

[LadyPropane]

Again issues with Anaconda.

I’m trying to install Qubes on another computer and I’m 100% sure I followed the steps correctly.

First, after clicking “Refresh”, to refresh the disks, Anaconda doesn’t show the USB drive. I put another one and it does show it. OK, perfect.

Then, I try to add a “EFI System” partition. I delete the old partition on the USB drive. It had some live Linux on it. Whatever. Then it asks me for a partition table, but it shouldn’t, right? So I dd the drive, reboot (just to have a clean slate) and try again.

I follow the steps and when I get to the Blivet GUI, it doesn’t ask me for a partition table this time. Perfect. I click to add a new partition and I see these options in the “Filesystem” field:

BIOS Boot
btrfs
ext2
ext3
ext4
physical volume (LVM)
ntfs
swap
vfat
xfs
unformatted

I don’t know why I can’t choose “EFI System”.

Should I format the drive before? What am I doing wrong? I’ve been at it for 2 hours now and I’m out of ideas.

It feels like it might be an issue with Anaconda. Just like last time (in my previous comments in this thread), I feel like this is going to resolve itself after the nth attempt, but I would like a real solution, because I am really annoyed right now.

The previous laptop I was installing Qubes to and this desktop have nothing in common. Different USB drives, as well. Yet the same issue with not being able to select “EFI System”.

[apparatus]

Maybe your laptop doesn’t support UEFI? Or you booted the Qubes OS installer in the Legacy BIOS mode?
You don’t need to create a EFI System partition for Legacy BIOS mode, you can skip it.

[LadyPropane]

Haha, that solved it. I booted by just selecting the “USB: SanDisk” option (or something like that), not the “UEFI: SanDisk, partition 2” (or something like that). I had to scroll down a bit to see that option, I feel so stupid. Thanks

[xpetrovich]

Hello!
I tried to install according to this instruction. The installation was successful, but the first time I run it, a sys-usb VM startup error occurs.
Also, after the first setup, qubes-os starts, but the sys-usb virtual machine still does not start.
If I reboot the PC, qubes-os does not start, but again requires initial setup.
Ideas?
Thanks in advance.

[apparatus]

It’s not related to this guide, I think you’d have the same error even with default Qubes OS installation.
Boot Qubes OS with disabled qubes autostart so sys-usb won’t start automatically like this:
Autostart troubleshooting | Qubes OS
When you boot in dom0 then try to start sys-usb and if you’ll have the same error then check the logs in dom0 to see the error:
/var/log/libvirt/libxl/libxl-driver.log
/var/log/xen/console/guest-sys-usb.log
/var/log/xen/console/guest-sys-usb-dm.log
And you can also use journalctl command in dom0 to check the dom0 logs.