Update
I have successfully installed Qubes OS onto a USB drive (it took ~12 hours with USB 2.0) after some troubleshooting. Here are some notes on my progress.
If any of you have the time and expertise to review this and pick it apart, I gladly welcome it.
Troubleshooting the intermediate installation
For the sake of documentation, I will mention that the first installation attempt onto the USB resulted in a “Pane is dead” error, and it seems to have occurred during installation of the bootloader (it froze there). When I attempted to boot anyway however, it booted fine until it tried to start up the Qubes OS daemon, at which point it hanged indefinitely (I let it run for over 30 minutes). After some reading on the topic, I decided to access the partial installation from the rescue disk to inspect what might be the problem. Seeing that qubesd.service points to /usr/bin/qubesd as the ExecStart, I checked for that file, only to discover it missing! So that is probably around where Pane actually died: installing qubes-core-dom, which contains qubesd.
During the second installation, which was a success (no dead Pane), I kept getting timed out waiting for the LUKS volume to load. Each time, the Dracut emergency shell arrives. After a few failed attempts, I decided to check /etc/fstab alongside lsblk, which revealed that the wrong UUID was included in fstab for the volume! I have no idea why, maybe it was from the previous failed installation, but regardless I manually corrected fstab and rebooted. The boot was successful and I was able to login.
Lessons: Don’t let Pane die and check /etc/fstab if loading the volume puts you in timeout.
Successfully running Qubes OS on USB
Once logged in, I initiated the installation of the initial VMs, but realized after some time that I will want to manually reinitialize all this once I am on the SSD regardless, so I killed the process. It was filling my poor little USB stick up to the maximum, anyway. Then, I cleaned up the mess I made by attempting to uninstall all the VMs, then manually uninstalling them after some errors. Once I have migrated to the SSD, I will run the following command to reinitialize the VM setup (source):
sudo /usr/libexec/initial-setup/initial-setup-graphical
Patching dracut
Next is to patch the crypt module of dracut to support using, among other things, the serial ID as an identifier through the undocumented rd.luks.serial parameter (thanks to this tip!). This feature was added after the version available in the default Fedora 25 VM on dom0, which is why it will need to be manually modified. Fortunately, the work was already done for us by the same privb0x23 whose article first confirmed my idea that this custom setup was possible (the “Gold Solution”); their merged pull request can be found here and these are the changes that need to be made. It’s a small patch to a few files, so I added it in by hand.
If you are not trying to air gap your target computer from your old one, to avoid transferring files over USB between them without USB qube protection (which is not feasible while running from a USB installation), or to keep your intermediate installation offline like I am, then you can just copy the files over or update dom0 over the network. Doing so will guarantee you won’t accidentally insert a bug during manual trans-coding.
Editing /etc
Now, to edit the relevant /etc files, which can be done before or after migration. If done before, they should be renamed or commented out to prevent breaking the intermediate installation in case of troubleshooting or problems during migration. This will assume it is done before.
First is /etc/crypttab.newinstall:
nano /etc/crypttab.newinstall
---
# encrypted /boot
cryptoboot /dev/disk/by-id/<USB-FD-ID> none luks
# first LUKS layer
luks0 /dev/disk/by-id/<ATA-SSD-ID> /boot/<KEYFILE> luks,noearly,header=/boot/<HEADER-0>,keyfile-offset=<OFFSET>,keyfile-size=<SIZE>
# second LUKS layer
luks1 /dev/mapper/luks0 /boot/<KEYFILE> luks,noearly,header=/boot/<HEADER-1>,keyfile-offset=<OFFSET>,keyfile-size=<SIZE>
# third LUKS layer
luks2 /dev/mapper/luks1 /boot/<KEYFILE> luks,noearly,header=/boot/<HEADER-2>,keyfile-offset=<OFFSET>,keyfile-size=<SIZE>
Then /etc/fstab.newinstall:
nano /etc/fstab.newinstall
---
/dev/mapper/cryptoboot /boot ext2 ro,noatime,noauto,nodev,noexec,nosuid
/dev/qubes_dom0/root / ext4 defaults
/dev/qubes_dom0/swap swap defaults
And then create a custom dracut configuration file:
nano /etc/dracut.conf.d/install-items.conf
---
#install_items+="/boot/<HEADER-0> /boot/<HEADER-1> /boot/<HEADER-2> /boot/<KEYFILE> /etc/crypttab"
(N.B. This will be uncommented after migration.)
Next, modifying /etc/default/grub.newinstall:
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="gfxterm"
GRUB_CMDLINE_LINUX="rd.luks.serial=<ata.SSD-ID> rd.lvm.lv=qubes_dom0/root rd.lvm.lv=qubes_dom0/swap i915.alpha_support=1 plymouth.ignore-serial-consoles rhgb quiet"
GRUB_CMDLINE_XEN_DEFAULT="console=none dom0_mem=min:1024M dom0_mem=max:4096M iomenu=no-igfx ucode=scan smt=off"
GRUB_DISABLE_RECOVERY="true"
GRUB_THEME="/boot/grub2/themes/system/theme.txt"
GRUB_DISABLE_OS_PROBER="true"
GRUB_ENABLE_CRYPTODISK=y
In preparation for the next step, edit the actual /etc/default/grub and add -old to the end of each LV:
... rd.lvm.lv=qubes_dom0/root rd.lvm.lv=qubes_dom0/swap ... --> ... rd.lvm.lv=qubes_dom0/root-old rd.lvm.lv=qubes_dom0/swap-old ...
Then, run grub-mkconfig -o /boot/grub2/grub.cfg to apply the configuration changes.
And finally, to avoid any LVM conflicts, rename the volume group and logical volumes:
lvrename qubes_dom0/root qubes_dom0/root-old
lvrename qubes_dom0/pool00 qubes_dom0/pool00-old
lvrename qubes_dom0/swap qubes_dom0/swap-old
vgrename qubes_dom0 qubes_dom0-old
Migrating from USB to SSD via rescue disk
Once all this is complete, I will then proceed with migrating the system. To do this, I will shutdown the computer and reboot from the rescue disk (sr0). I will then attach the USB detached /boot drive (sdb) and the USB Qubes installation (sdc), enter the shell, and proceed with the following:
# make directories to house USB (sdc) and SSD (sda) Qubes mounts
mkdir /mnt/usb-qubes
mkdir /mnt/ssd-qubes
# open and mount LUKS volume for USB detached /boot drive (sdb)
# mount read-only for this step as it won't be written to
cryptsetup -v luksOpen /dev/sdb1 cryptoboot
mount -o ro /dev/mapper/cryptoboot /mnt
# open and activate LUKS and LVM volumes for SSD Qubes
cryptsetup -v luksOpen --header /mnt/<HEADER-0> --key-file=<KEYFILE> --keyfile-offset=<OFFSET> --keyfile-size=<SIZE> /dev/sda luks0
cryptsetup -v luksOpen --header /mnt/<HEADER-1> --key-file=<KEYFILE> --keyfile-offset=<OFFSET> --keyfile-size=<SIZE> /dev/mapper/luks0 luks1
cryptsetup -v luksOpen --header /mnt/<HEADER-2> --key-file=<KEYFILE> --keyfile-offset=<OFFSET> --keyfile-size=<SIZE> /dev/mapper/luks1 luks2
vgchange -ay qubes_dom0
# mount SSD Qubes
mount /dev/qubes_dom0/root /mnt/ssd-qubes
# open and activate LUKS and LVM volumes on USB Qubes
cryptsetup -v luksOpen /dev/sdc2 usb-luks
vgchange -ay qubes_dom0-old
# mount USB Qubes, read-only to prevent any disasters and
# because nothing will be written to it at this point
mount -o ro /dev/qubes_dom0-old/root /mnt/usb-qubes
# unmount USB detached /boot drive (sdb), as it is not needed for now
umount /dev/mapper/cryptoboot
# migrate the root filesystem, preserving all information and
# avoiding fragmentation, /boot will come later; this will
# take a very long time
rsync -aAXHxW --info=progress2 --preallocate --exclude={"/boot/*","/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found"} /mnt/usb-qubes /mnt/ssd-qubes && sync
# mount the USB Qubes's /boot (sdc1) and
# remount the USB detached /boot drive
mount /dev/sdc1 /mnt/usb-qubes/boot
mount /dev/mapper/cryptoboot /mnt/ssd-qubes/boot
# migrate /boot like before, but without excluding anything and to the /mnt/*/boot mountpoints
rsync -aAXHxW --info=progress2 --preallocate /mnt/usb-qubes/boot /mnt/ata-qubes/boot && sync
# unmount the USB Qubes and /boot, as it is no longer needed
umount /mnt/usb-qubes/boot
umount /mnt/usb-qubes
# mount in preparation of chrooting into SSD Qubes
mount -t proc none /mnt/ssd-qubes/proc
mount -o bind /dev /mnt/ssd-qubes/dev
mount -o bind /sys /mnt/ssd-qubes/sys
mount -o bind /run /mnt/ssd-qubes/run
# chroot into the SSD Qubes
chroot /mnt/ssd-qubes
# fix /etc by removing old files and renaming new ones
rm /etc/fstab /etc/crypttab /etc/default/grub
rename -v fstab.newinstall fstab /etc/fstab.newinstall
rename -v crypttab.newinstall crypttab /etc/crypttab.newinstall
rename -v grub.newinstall grub /etc/default/grub.newinstall
echo 'install_items+="/boot/<HEADER-0> /boot/<HEADER-1> /boot/<HEADER-2> /boot/<KEYFILE> /etc/crypttab"' > /etc/dracut.conf.d/install-items.conf
# generate new initramfs
dracut --force
# reconfigure and install GRUB
grub-mkconfig -o /boot/grub2/grub.cfg
grub-install /dev/sdb
# clean up - exit chroot, unmount SSD Qubes and USB detached /boot drive
exit
umount /mnt/ssd-qubes/boot
umount /mnt/ssd-qubes
# shut down computer
shutdown now
Once this is done, I think I should be able to boot the newly-migrated Qubes OS on the SSD.
Problems
This plan is a patch-work of many diverse sources, many of which contradict each other and almost none of which has anything to do with implementing what I am trying to do. I suspect that there is some overlap or redundancy in this plan that could be trimmed out, but I am not sure where. I am basically throwing whatever sticks. Any advice from the shrewd eyes here will be much appreciated.
I also am worried that this may not be sufficient due to the apparent lack of support by systemd, in which case I may need to make a copy of this as /etc/crypttab.initramfs. However, since I installed /etc/crypttab into the new initramfs through the custom dracut configuration file, and since dracut handles initramfs on Fedora 25, perhaps this won’t be necessary.
Lastly, I have a lingering feeling that I am missing something here, something important.
I hope that you all can critique my plan and point out its flaws so that I don’t royally screw up my migration. If any of you know what that missing something might be (other than nothing), I very much would like to know too.
Thoughts?
All the best,
John