Intermediate install of Qubes OS on USB

Hello Qubes OS community,

This is the fourth and hopefully final topic in my personal saga of installing Qubes OS inside LVM on LUKS in a triple cascade with detached encrypted /boot and LUKS headers and without partition table in order to achieve deniable full-disk cascading encryption with two-factor authentication. (Phew, quite a mouthful!) As I said before, if successful, I intend to write a guide for anyone crazy enough to follow it to implement this as well, including through an automated Kickstart file if at all possible.

After first exploring the possibility of a partitionless installation and wrapping my head around implementing cascading encryption with dm-crypt+LUKS, only to find that such an implementation may prove to be impossible to install on block devices without partition tables after all due to the limitations of Anaconda (with a brief excursion into Ask Fedora that only confirmed my suspicions and suggested distant hope of a bespoke Kickstart solution), I am now resigned to an intermediate installation path to just finally get me running Qubes OS.

This intermediate installation will be more or less standard, but onto a USB stick (sdc) whose data will then be cp'd or rsync'd onto the original main drive sda (with /boot encrypted onto USB drive sdb). This is not what I wanted and this should not be necessary, but this is the price of the custom installation I want due to the constraints of Anaconda, and it is a method that others have used for other distros whenever met with similar obstacles.

So, it should be a rather straightforward procedure, at least once I figure out what exactly that procedure is. :sweat_smile: My question to you, Qubes community, is whether there is anything specific to Qubes OS that might serve as a pitfall during this process, especially given that Qubes OS is by no means a typical Linux distro.

For example, will Xen play nice with this plan, including being kept on the separate /boot drive (which is where I assume it will reside)? Also, I understand that I will probably need to modify some of the files during the process, such as GRUB (which will also need to be reinstalled), crypttab, and fstab. Is there anything else I will need to modify, or otherwise cleanup as remnants of the intermediate install? Is there anything else I should watch out for, or advice I should heed, during this process?

Many thanks,
John

2 Likes

I’ll admit I just glanced over your older posts so if I ask something you already answered I do apologize:

  • why not just use plain dm-crypt for the first layer of encryption? iirc grub modifications needed are fairly straightforward, you can use a relatively easy password or even save the decryption key plaintext to the trusted boot-usb; the data on your disks looks random, and you can nest luks containers underneath it to have some proper encryption

  • alternatively why not do a straight up normal installation of qubes, let it format and encrypt the disk however it wants, then reboot, copy /boot on usb, copy/recreate the header on usb, overwrite the /boot, luks header and partition table on the hard disk ? you’d have to change the offsets in luks header but I think that’d be it?

Honestly I probably won’t be of much help to you at the end and this seems important to you, so feel free to ignore this post and spend your time looking for people with an actual clue :slight_smile:

1 Like

Hello @billystonka,

I appreciate the interest. I’ll try to respond to your points to the best of my abilities, and invite any further questions or criticism you and everyone else may have about it.

I chose LUKS over plain dm-crypt because I wanted to take advantage of the salting and key stretching that LUKS provides, though their benefits may not appreciably impact my security margin due to how absurdly long and complex my passphrases are. In fact, the existence of the LUKS master key may actually be more of a weak point because its entropy may be lower than my /boot LUKS passphrase (which is >256 bits of entropy) and of the key files (>4096 bytes of /dev/urandom input each), and because its loss due to corruption or erasure would render everything irrecoverable regardless of whether I provided correct passphrases and key files. Nonetheless, on the assumption that I am mistaken due to my relative unfamiliarity with how dm-crypt works internally, and because I thought it would simplify my setup rather than complicate it (plus more documentation), I opted for LUKS.

Had password hashing and salting been implemented in plain dm-crypt, I would have probably used plain dm-crypt, since the hashing would have directly impacted the strength of the “master key” and not just the passphrase that encrypts it.

Plain dm-crypt may make more sense for the USB boot drive sdb, and I am still considering that option, but I am under the impression that the most achievable without secure boot is a minimally exposed MBR partition table or cleartext bootloader, since a totally encrypted setup is not possible, at least not without a custom TPM-enabled secure boot with hardware-based FDE or something similar (and even then I’m not sure). So the only benefit that dm-crypt provides there is avoiding a cleartext LUKS header (thus obfuscating the encryption of sdb). I would love to be wrong about this, since finally encrypting the partition table (or doing away with it altogether) is a solution I have been searching for unsuccessfully for a while now. It probably does exist or is possible, I just lack the technical expertise to understand it.

Admittedly, I also still struggle to fully understand dm-crypt and lack sufficient knowledge to confidently determine which encryption method would afford a stronger security margin (LUKS or plain, assuming passphrases/key files >256 bits of entropy), so my analysis above may be entirely mistaken and fails to fully appreciate the role of the master key in LUKS. Clarification on this matter would be much appreciated, but I still intend to be implementing LUKS all the way down this time around, at least until I have to reinstall Qubes OS and get to re-experience the “joy” of reimplementing this and deciding how I might tweak it further. Hopefully by then, LUKS2 with Argon2, integrity/authentication, and maybe even SHA3 or BLAKE2 will be available for Qubes OS installation.

As for using plain dm-crypt for the first layer of sda encryption, the benefit is unclear to me. I may be misunderstanding you, but it seems to me that I would have to use plain dm-crypt as a substitute for all my LUKS encryption, since otherwise I will be handling detached headers at some point anyway and so one more as the first layer is not significant. Are you saying that the first plain dm-crypt layer on sda would be easier to hook into /boot through GRUB2, as opposed to the more difficult method involving a detached header?

If I correctly understand your second point, then it is because I would have to repartition sda to reclaim the /boot partition space on sda1, which comes with its own headache with expanding/re-encrypting the LUKS containers to reclaim/encrypt the added space; and it carries more inherent risk of remnant cleartext on sda, especially due to the special properties of SSDs; all while still including the same custom initramfs/dracut problems that I will be facing anyway. By comparison, an intermediate install path does not suffer from the repartitioning problem or remnant cleartext risk, since no cleartext has ever touched this SSD so far and it is already partitioned in the way I want it, ready for the files to simply be copied over.

If I misunderstood you or the tools/methods/concepts involved at any point, please correct me where I am wrong. This whole process has been a humbling learning experience and I am open to learning some more.

All the best,
John

1 Like

Update

I have successfully installed Qubes OS onto a USB drive (it took ~12 hours with USB 2.0) after some troubleshooting. Here are some notes on my progress.

If any of you have the time and expertise to review this and pick it apart, I gladly welcome it.

Troubleshooting the intermediate installation

For the sake of documentation, I will mention that the first installation attempt onto the USB resulted in a “Pane is dead” error, and it seems to have occurred during installation of the bootloader (it froze there). When I attempted to boot anyway however, it booted fine until it tried to start up the Qubes OS daemon, at which point it hanged indefinitely (I let it run for over 30 minutes). After some reading on the topic, I decided to access the partial installation from the rescue disk to inspect what might be the problem. Seeing that qubesd.service points to /usr/bin/qubesd as the ExecStart, I checked for that file, only to discover it missing! So that is probably around where Pane actually died: installing qubes-core-dom, which contains qubesd.

During the second installation, which was a success (no dead Pane), I kept getting timed out waiting for the LUKS volume to load. Each time, the Dracut emergency shell arrives. After a few failed attempts, I decided to check /etc/fstab alongside lsblk, which revealed that the wrong UUID was included in fstab for the volume! I have no idea why, maybe it was from the previous failed installation, but regardless I manually corrected fstab and rebooted. The boot was successful and I was able to login.

Lessons: Don’t let Pane die and check /etc/fstab if loading the volume puts you in timeout.

Successfully running Qubes OS on USB

Once logged in, I initiated the installation of the initial VMs, but realized after some time that I will want to manually reinitialize all this once I am on the SSD regardless, so I killed the process. It was filling my poor little USB stick up to the maximum, anyway. Then, I cleaned up the mess I made by attempting to uninstall all the VMs, then manually uninstalling them after some errors. Once I have migrated to the SSD, I will run the following command to reinitialize the VM setup (source):

sudo /usr/libexec/initial-setup/initial-setup-graphical

Patching dracut

Next is to patch the crypt module of dracut to support using, among other things, the serial ID as an identifier through the undocumented rd.luks.serial parameter (thanks to this tip!). This feature was added after the version available in the default Fedora 25 VM on dom0, which is why it will need to be manually modified. Fortunately, the work was already done for us by the same privb0x23 whose article first confirmed my idea that this custom setup was possible (the “Gold Solution”); their merged pull request can be found here and these are the changes that need to be made. It’s a small patch to a few files, so I added it in by hand.

If you are not trying to air gap your target computer from your old one, to avoid transferring files over USB between them without USB qube protection (which is not feasible while running from a USB installation), or to keep your intermediate installation offline like I am, then you can just copy the files over or update dom0 over the network. Doing so will guarantee you won’t accidentally insert a bug during manual trans-coding.

Editing /etc

Now, to edit the relevant /etc files, which can be done before or after migration. If done before, they should be renamed or commented out to prevent breaking the intermediate installation in case of troubleshooting or problems during migration. This will assume it is done before.

First is /etc/crypttab.newinstall:

nano /etc/crypttab.newinstall
---
# encrypted /boot
cryptoboot   /dev/disk/by-id/<USB-FD-ID>    none              luks

# first LUKS layer
luks0        /dev/disk/by-id/<ATA-SSD-ID>   /boot/<KEYFILE>   luks,noearly,header=/boot/<HEADER-0>,keyfile-offset=<OFFSET>,keyfile-size=<SIZE>

# second LUKS layer
luks1        /dev/mapper/luks0              /boot/<KEYFILE>   luks,noearly,header=/boot/<HEADER-1>,keyfile-offset=<OFFSET>,keyfile-size=<SIZE>

# third LUKS layer
luks2        /dev/mapper/luks1              /boot/<KEYFILE>   luks,noearly,header=/boot/<HEADER-2>,keyfile-offset=<OFFSET>,keyfile-size=<SIZE>

Then /etc/fstab.newinstall:

nano /etc/fstab.newinstall
---
/dev/mapper/cryptoboot   /boot   ext2   ro,noatime,noauto,nodev,noexec,nosuid
/dev/qubes_dom0/root     /       ext4   defaults
/dev/qubes_dom0/swap             swap   defaults

And then create a custom dracut configuration file:

nano /etc/dracut.conf.d/install-items.conf
---
#install_items+="/boot/<HEADER-0> /boot/<HEADER-1> /boot/<HEADER-2> /boot/<KEYFILE> /etc/crypttab"

(N.B. This will be uncommented after migration.)

Next, modifying /etc/default/grub.newinstall:

GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="gfxterm"
GRUB_CMDLINE_LINUX="rd.luks.serial=<ata.SSD-ID> rd.lvm.lv=qubes_dom0/root rd.lvm.lv=qubes_dom0/swap i915.alpha_support=1 plymouth.ignore-serial-consoles rhgb quiet"
GRUB_CMDLINE_XEN_DEFAULT="console=none dom0_mem=min:1024M dom0_mem=max:4096M iomenu=no-igfx ucode=scan smt=off"
GRUB_DISABLE_RECOVERY="true"
GRUB_THEME="/boot/grub2/themes/system/theme.txt"
GRUB_DISABLE_OS_PROBER="true"
GRUB_ENABLE_CRYPTODISK=y

In preparation for the next step, edit the actual /etc/default/grub and add -old to the end of each LV:

... rd.lvm.lv=qubes_dom0/root rd.lvm.lv=qubes_dom0/swap ... --> ... rd.lvm.lv=qubes_dom0/root-old rd.lvm.lv=qubes_dom0/swap-old ...

Then, run grub-mkconfig -o /boot/grub2/grub.cfg to apply the configuration changes.

And finally, to avoid any LVM conflicts, rename the volume group and logical volumes:

lvrename qubes_dom0/root qubes_dom0/root-old
lvrename qubes_dom0/pool00 qubes_dom0/pool00-old
lvrename qubes_dom0/swap qubes_dom0/swap-old
vgrename qubes_dom0 qubes_dom0-old

Migrating from USB to SSD via rescue disk

Once all this is complete, I will then proceed with migrating the system. To do this, I will shutdown the computer and reboot from the rescue disk (sr0). I will then attach the USB detached /boot drive (sdb) and the USB Qubes installation (sdc), enter the shell, and proceed with the following:

# make directories to house USB (sdc) and SSD (sda) Qubes mounts
mkdir /mnt/usb-qubes
mkdir /mnt/ssd-qubes

# open and mount LUKS volume for USB detached /boot drive (sdb)
# mount read-only for this step as it won't be written to
cryptsetup -v luksOpen /dev/sdb1 cryptoboot
mount -o ro /dev/mapper/cryptoboot /mnt

# open and activate LUKS and LVM volumes for SSD Qubes
cryptsetup -v luksOpen --header /mnt/<HEADER-0> --key-file=<KEYFILE> --keyfile-offset=<OFFSET> --keyfile-size=<SIZE> /dev/sda luks0
cryptsetup -v luksOpen --header /mnt/<HEADER-1> --key-file=<KEYFILE> --keyfile-offset=<OFFSET> --keyfile-size=<SIZE> /dev/mapper/luks0 luks1
cryptsetup -v luksOpen --header /mnt/<HEADER-2> --key-file=<KEYFILE> --keyfile-offset=<OFFSET> --keyfile-size=<SIZE> /dev/mapper/luks1 luks2
vgchange -ay qubes_dom0

# mount SSD Qubes
mount /dev/qubes_dom0/root /mnt/ssd-qubes

# open and activate LUKS and LVM volumes on USB Qubes
cryptsetup -v luksOpen /dev/sdc2 usb-luks
vgchange -ay qubes_dom0-old

# mount USB Qubes, read-only to prevent any disasters and
# because nothing will be written to it at this point
mount -o ro /dev/qubes_dom0-old/root /mnt/usb-qubes

# unmount USB detached /boot drive (sdb), as it is not needed for now
umount /dev/mapper/cryptoboot

# migrate the root filesystem, preserving all information and
# avoiding fragmentation, /boot will come later; this will
# take a very long time
rsync -aAXHxW --info=progress2 --preallocate --exclude={"/boot/*","/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found"} /mnt/usb-qubes /mnt/ssd-qubes && sync

# mount the USB Qubes's /boot (sdc1) and
# remount the USB detached /boot drive
mount /dev/sdc1 /mnt/usb-qubes/boot
mount /dev/mapper/cryptoboot /mnt/ssd-qubes/boot

# migrate /boot like before, but without excluding anything and to the /mnt/*/boot mountpoints
rsync -aAXHxW --info=progress2 --preallocate /mnt/usb-qubes/boot /mnt/ata-qubes/boot && sync

# unmount the USB Qubes and /boot, as it is no longer needed
umount /mnt/usb-qubes/boot
umount /mnt/usb-qubes

# mount in preparation of chrooting into SSD Qubes
mount -t proc none /mnt/ssd-qubes/proc
mount -o bind /dev /mnt/ssd-qubes/dev
mount -o bind /sys /mnt/ssd-qubes/sys
mount -o bind /run /mnt/ssd-qubes/run

# chroot into the SSD Qubes
chroot /mnt/ssd-qubes

# fix /etc by removing old files and renaming new ones
rm /etc/fstab /etc/crypttab /etc/default/grub
rename -v fstab.newinstall fstab /etc/fstab.newinstall
rename -v crypttab.newinstall crypttab /etc/crypttab.newinstall
rename -v grub.newinstall grub /etc/default/grub.newinstall
echo 'install_items+="/boot/<HEADER-0> /boot/<HEADER-1> /boot/<HEADER-2> /boot/<KEYFILE> /etc/crypttab"' > /etc/dracut.conf.d/install-items.conf

# generate new initramfs
dracut --force

# reconfigure and install GRUB
grub-mkconfig -o /boot/grub2/grub.cfg
grub-install /dev/sdb

# clean up - exit chroot, unmount SSD Qubes and USB detached /boot drive
exit
umount /mnt/ssd-qubes/boot
umount /mnt/ssd-qubes

# shut down computer
shutdown now

Once this is done, I think I should be able to boot the newly-migrated Qubes OS on the SSD.

Problems

This plan is a patch-work of many diverse sources, many of which contradict each other and almost none of which has anything to do with implementing what I am trying to do. I suspect that there is some overlap or redundancy in this plan that could be trimmed out, but I am not sure where. I am basically throwing whatever sticks. Any advice from the shrewd eyes here will be much appreciated.

I also am worried that this may not be sufficient due to the apparent lack of support by systemd, in which case I may need to make a copy of this as /etc/crypttab.initramfs. However, since I installed /etc/crypttab into the new initramfs through the custom dracut configuration file, and since dracut handles initramfs on Fedora 25, perhaps this won’t be necessary.

Lastly, I have a lingering feeling that I am missing something here, something important.

I hope that you all can critique my plan and point out its flaws so that I don’t royally screw up my migration. If any of you know what that missing something might be (other than nothing), I very much would like to know too.

Thoughts?

All the best,
John

2 Likes

Hello John, nice to see you have made progress!

Sadly, as I expected, I haven’t even properly understood your goals (still not sure if I do!) and therefore might’ve just wasted your time with my post. I’ll try to clarify what I meant, but again, you are probably better off spending your time on literally anything else :slight_smile: I’d not even bother replying if not for the fact that I used to have a debian machine set up in a way similar to what (I think) you are trying to achieve, but I don’t remember having that much troubles setting it up (can’t remember any actually) so I suspect what I ran was much more primitive than what you are trying to do.

The “insecurity” of dm-crypt doesn’t really matter here as you’d only be using it for the very first layer and strictly to have achieve “”“plausible deniability”“” thanks to lack of any headers. You can use an easy password or a keyfile from the “secure” booting USB drive and have as many nested LUKS containers encapsulated in it as you want. However, in your case it doesn’t matter since you plan to keep all your headers on a single encrypted GRUB USB, which I addressed below:

Omg I was under the impression that each of the LUKS layers will also be “hidden” from the upper layers by detaching/hiding headers so you can achieve multiple layers of plausible deniability! But in fact the sole reason you are even nesting LUKS containers is to have the data encrypted with multiple ciphers?

Does this mean that this whole lengthy process is exclusively to work around Anaconda refusing to install to partitionless disk?!

If that’s the case I hope you don’t mind me asking following questions (I am sorry if you addressed any of this before, I skimmed through all your old posts I was able to find, but I might’ve missed ton of stuff):

  • have you considered or maybe even tried modifying Anaconda? it’s written in easily readable Python. Might be worth checking if things won’t work if you simply comment out some of the checks :)? You probably wouldn’t even need to patch the iso just edit .py files from console when you boot the installer. I know this might sound a little primitive, but it might just work and would save you a ton of work here…

  • even before patching anything, why not switch to console after the installer starts, format and map as many layers of encryption as you can be bothered to and on the final layer initialize a LVM? At this point I think Anaconda (and any installer really) should be willing to partition it for you, but if it doesn’t you could still partition it by hand? At which point the installation should proceed as normal and if it doesn’t there is a serious bug somewhere.

If either of those work then all that’d be left to do is live boot, open all the containers by hand, chroot and enable encryption in grub and adjust the scripts so all the containers get opened and mapped before you try to mount root. Copy /boot to USB. Done. The only downside I can think of would be the redundant /boot partition in your LVM, but that’s nothing unless you are VERY pressed for space. The whole process should not extend the standard installation process by more than 30 minutes…

Of course I imagine you’d try all of those things before you started with this more complex approach of yours, which again, most likely means I am still not understanding what you are actually trying to accomplish. Hopefully I can serve as a rubber ducky at the very least :slight_smile:

I realized it’d be rude not to answer some of your questions you addressed to me.

DISCLAIMER: anybody that treats anything I say about cryptography seriously shouldn’t be allowed to operate a computer !!!

As far as I can understand LUKS vs dm-crypt, the only benefit you’d get out of LUKS is the possibility to add/change/revoke passwords for any particular layer. However from how you described your setup and since you mentioned worrying about data retention on a medium, I’m gonna go ahead and guess that if you ever need to change any of the passphrases you’d rather make a whole new container with a whole new header and so the only (in your case) advantage of LUKS is gone.

All the hashing and stretching and whatnot is implemented to (afaik) get a like-random master key (max size 512 bits, default 256 bits) from a probably-not-so-random user passphrase (or a keyfile). Since in your case you will be using random (or like-random) passphrases/keyfiles, you don’t really benefit from any of that. If truly random, you could use the first 512 (or 256) bits of your keyfile directly as your master key and you would not (afaik) lose any security except for operational security (which I think you mentioned is not a concern here).

The most obvious benefit for me would be the ability to access anything under the first layer of encryption with just a passphrase alone (no need for the USB drive or the headers) while maintaining plausible deniability. You can keep some helper scripts and non critical data in the first layer next to the nested containers (by using partitions or offsets).
Since you will be opening all the containers from the same place (scripts/configs on GRUB USB) it doesn’t really matter if you use dmcrypt or detached headers or any kind of mix of those, the level of security should stay the same. Neither one should be harder to handle than the other.

Since the USB must be kept secure and you can’t really achieve any kind of plausible deniability with it, it’s the one place where having a traditional LUKS container with the headers attached would make the most sense in my eyes :slight_smile:

If you worry about remnants of cleartext you can use an intermediate, disposable SSD/USB a bit like you are doing. Except instead of recreating the LVM and all the partitions and copying data by hand with rsync for some reason you could just dump the whole binary image of the intermediate disk to the permanent one, sans the cleartext beginning. You’ll have to manually change the offset to 0 in LUKS header and few other things GRUB but that should be about it.

And some additional things:

  • can you not address your FDEd disk from grub with /dev/disk/by-id/ ?
  • AFAIK out of the box LUKS uses either a passphrase OR a keyfile but not both at once, are you ok with that?