Playing with qubes part 2

Community Guides
1

Part 2 of Playing with qubes

Is a rewrite of my old notes, Qubes OS Installation - Detached encrypted boot and header

Thank you @kaizer and @dum0 for donating to qubes so that I could rewrite this guide.

I’d recommend that you try in VM first, before doing it on your machine, so that you can also learn and understand too.

This is UEFI based only.

  • Prerequisite :

    • QubesOS Installation Medium.
    • 2 Drives for separated Root and Boot/EFI Partition, and we would call them with :
      • Root = /dev/nvme0n1
      • Boot = /dev/sda

  • After booting into installation in language section, press ctrl + alt + f2 to enter tty2

  • Format drive we would use for the installation.

    • dd if=/dev/urandom of=/dev/nvme0n1 bs=1M status=progress

    • dd if=/dev/urandom of=/dev/sda bs=1M status=progress

  • Create Boot, EFI, Root Header, Swap Header Partition.

    Boot Partition

    • sgdisk -n 0:0:+200MiB -t 0:ef02 /dev/sda

    EFI Partition

    • sgdisk -n 0:0:+200MiB -t 0:ef00 /dev/sda

    Root Header Partition

    • sgdisk -n 0:0:+16MiB -t 0:8309 /dev/sda

    If you want to use BTRFS also add swap header, another FS is not needed.

    Swap Header Partition

    • sgdisk -n 0:0:+16MiB -t 0:8309 /dev/sda

XFS / EXT4

  • Create custom Luks configuration.

    • cryptsetup -c aes-xts-plain64 -h sha512 -s 512 -y -i 10000 luksFormat /dev/nvme0n1

    You may want to use -i 1 (iterations 1ms) to speed up decrypting process for practice.

    Of course you can choose another -c (chiper) -h (hash size) -s (key size)

    • cryptsetup luksOpen /dev/nvme0n1 luks-root

    • pvcreate /dev/mapper/luks-root

    • vgcreate qubes_dom0 /dev/mapper/luks-root

    • lvcreate -L 8G -n swap qubes_dom0

    You might want to read 8.2. Recommended system swap space if you’re not sure about swap space

    • lvcreate -T -L 40G qubes_dom0/root-pool

    • lvcreate -T -l +90%FREE qubes_dom0/vm-pool

    SSD would perform badly if all of the space is used, so we leave 10% of the total to maintain the best performance.

    • lvcreate -V30G -T qubes_dom0/root-pool -n root-pool

    You should leave at least 10% in total of Thin Pool space so it could tell you if your drive will out of space.

    • lvs

    To see how much space you have in the VM pool, and use it to create VM lv.

    • lvcreate -V800G -T qubes_dom0/vm-pool -n vm

    XFS

    • mkfs.xfs /dev/qubes_dom0/vm

    • mkswap /dev/qubes_dom0/swap

    EXT4

    • mkfs.ext4 /dev/qubes_dom0/vm

    • mkswap /dev/qubes_dom0/swap

BTRFS

  • Create Root and Swap Partition.

    Swap Partition

    • sgdisk -n 0:0:+8GiB -t 0:8200 /dev/nvme0n1

    Root Partition

    • sgdisk -n 0:0:0 -t 0:8304 /dev/nvme0n1

  • Create custom Luks configuration.

    • cryptsetup -c serpent-xts-plain64 -h sha512 -s 512 -y -i 10000 luksFormat /dev/nvme0n1p1

    • cryptsetup -c aes-xts-plain64 -h sha512 -s 512 -y -i 10000 luksFormat /dev/nvme0n1p2

    • cryptsetup luksOpen /dev/nvme0n1p1 luks-root

    • cryptsetup luksOpen /dev/nvme0n1p2 luks-swap

    • mkfs.btrfs --csum blake2b -L qubes_dom0 -d single /dev/mapper/luks-root

    You can use another -csum (checksum) and may want to read this if you want to learn another Checksum Algorithms

    • mkswap /dev/mapper/swap

Proceed GUI Installation

  • This is also a full video of this guide.

qubes

Begin installation.

  • After installation go back to tty2

    • cp -r /usr/lib/grub/x86_64-efi /mnt/sysroot/boot/efi/EFI/qubes/

    • chroot /mnt/sysroot/

    • mount -oremount,ro /boot

    • install -m0600 /dev/null /tmp/boot.tar

    • tar -C /boot --acls --xattrs --one-file-system -cf /tmp/boot.tar .

    • umount /boot/efi

    • umount /boot

  • Reformat Boot Partition

    • cryptsetup -c twofish-xts-plain64 -h sha512 -s 512 -y -i 1 --use-random --type luks1 luksFormat /dev/sda1

  • Create Alias

    uuidB = /boot partition

    uuidR = /root partition

    uuidS = swap partition

    • uuidB=“$(blkid -o value -s UUID /dev/sda1)”

    XFS / EXT4

    • uuidR=“$(blkid -o value -s UUID /dev/nvme0n1)”

    BTRFS

    • uuidS="$(blkid -o value -s UUID /dev/nvme0n1p1)

    • uuidR=“$(blkid -o value -s UUID /dev/nvme0n1p2)”

  • Reopen luks boot partition

    • cryptsetup luksOpen /dev/sda1 luks-$uuidB

    XFS

    • mkfs.xfs /dev/mapper/luks-$uuidB

    • xfs_admin -U $uuidB /dev/mapper/luks-$uuidB

    EXT4 / BTRFS

    • mkfs.ext2 -m0 -U $uuidB /dev/mapper/luks-$uuidB

  • Configure fstab

    • sed -i ‘s?UUID=QUBE[^ ]*?/dev/mapper/luks-’$uuidB’?g’ /etc/fstab

    Replace QUBE with the First 4 UUID Numbers of /dev/sda1

    XFS / EXT4

    • sed -i ‘s?/dev/qubes-dom0-root?/dev/mapper/luks-’$uuidR’?g’ /etc/fstab

    BTRFS

    • sed -i ‘s?UUID=QUBE[^ ]*?/dev/mapper/luks-’$uuidR’?g’ /etc/fstab

    • echo -e “/dev/mapper/luks-$uuidS none swap defaults 0 0” >> /etc/fstab

    Remount Boot and EFI partition

    • mount -v /boot

    • tar -C /boot --acls --xattrs -xf /tmp/boot.tar

    • mount /dev/sda1 /boot/efi

  • Configure keys

    • mkdir -m0700 /etc/keys

    • ( umask 0077 && dd if=/dev/urandom bs=1 count=64 of=/etc/keys/root.key conv=excl,fsync )

    • ( umask 0077 && dd if=/dev/urandom bs=1 count=64 of=/etc/keys/boot.key conv=excl,fsync )

    BTRFS also create swap key

    • ( umask 0077 && dd if=/dev/urandom bs=1 count=64 of=/etc/keys/swap.key conv=excl,fsync )

  • Configure LUKS for XFS / EXT4

    • cryptsetup luksAddKey /dev/nvme0n1 /etc/keys/root.key

    • cryptsetup luksAddKey /dev/sda1 /etc/keys/boot.key

    • cryptsetup luksHeaderBackup /dev/nvme0n1 --header-backup-file header

    • dd if=/header of=/dev/sda3 bs=16M count=1 status=progress

  • Configure LUKS for BTRFS

    • cryptsetup luksAddKey /dev/nvme0n1p2 /etc/keys/root.key

    • cryptsetup luksAddKey /dev/nvme0n1p1 /etc/keys/swap.key

    • cryptsetup luksAddKey /dev/sda1 /etc/keys/boot.key

    • cryptsetup luksHeaderBackup /dev/nvme0n1p2 --header-backup-file root-header

    • cryptsetup luksHeaderBackup /dev/nvme0n1p1 --header-backup-file swap-header

    • dd if=/root-header of=/dev/sda3 bs=16M count=1 status=progress

    • dd if=/swap-header of=/dev/sda4 bs=16M count=1 status=progress

  • Remove unnecessary files

    • shred -uvz /header

    • shred -uvz /tmp/boot.tar

  • Configure Crypttab

    XFS / EXT4

    • echo -e “luks-$uuidR /dev/nvme0n1 /etc/keys/root.key luks,discard,key-slot=1,header=/dev/sda3\nluks-$uuidB UUID=$uuidB /etc/keys/boot.key luks,key-slot=1” > /etc/crypttab

    BTRFS

    • echo -e “luks-$uuidR /dev/nvme0n1p2 /etc/keys/root.key luks,discard,key-slot=1,header=/dev/sda3\nluks-$uuidS /dev/nvme0n1p1 /etc/keys/swap.key luks,key-slot=1,header=/dev/sda4\nluks-$uuidB UUID=$uuidB /etc/keys/boot.key luks,key-slot=1” > /etc/crypttab

  • Configure GRUB

    • echo “GRUB_ENABLE_CRYPTODISK=y” >> /etc/default/grub

    • grub2-mkconfig -o /boot/efi/EFI/qubes.cfg

  • Rewrite uuid map and crypttab

    XFS / EXT4

    • sed -i ‘s?block_uuid.map"?block_uuid.map"\necho “/dev/nvme0n1 ‘$uuidR’\n/dev/disk/by-uuid/’$uuidB’ ‘$uuidB’" > “${initdir}/etc/block_uuid.map”?g’ /usr/lib/dracut/modules.d/90crypt/module-setup.sh

    • sed -i ‘s?$initdir/etc/crypttab?$initdir/etc/crypttab\necho “luks-’$uuidR’ /dev/nvme0n1 /etc/keys/root.key luks,discard,key-slot=1,header=/dev/sda3\nluks-‘$uuidB’ UUID=‘$uuidB’ /etc/keys/boot.key luks,key-slot=1” > $initdir/etc/crypttab?g’ /usr/lib/dracut/modules.d/90crypt/module-setup.sh

    BTRFS

    • sed -i ‘s?block_uuid.map"?block_uuid.map"\necho “/dev/nvme0n1p2 ‘$uuidR’\n/dev/nvme0n1p1 ‘$uuidS’\n/dev/disk/by-uuid/’$uuidB’ ‘$uuidB’" > “${initdir}/etc/block_uuid.map”?g’ /usr/lib/dracut/modules.d/90crypt/module-setup.sh

    • sed -i ‘s?$initdir/etc/crypttab?$initdir/etc/crypttab\necho “luks-’$uuidR’ /dev/nvme0n1p2 /etc/keys/root.key luks,discard,key-slot=1,header=/dev/sda3\nluks-‘$uuidS’ /dev/nvme0n1p1 /etc/keys/swap.key luks,discard,key-slot=1,header=/dev/sda4\nluks-‘$uuidB’ UUID=‘$uuidB’ /etc/keys/boot.key luks,key-slot=1” > $initdir/etc/crypttab?g’ /usr/lib/dracut/modules.d/90crypt/module-setup.sh

  • Configure Dracut

    • echo -e ‘add_dracutmodules+=" crypt “\ninstall_items+=” /etc/keys/root.key /etc/keys/boot.key ”’ > /etc/dracut.conf.d/qubes.conf

    • dracut -vf /boot/initramfs-*

    • exit

    • umount /mnt/sysroot/boot/efi

    • umount /mnt/sysroot/boot

    • umount -l /mnt/sysroot

    • umount -l /mnt/sysimage

    XFS / EXT4

    • swapoff /dev/qubes_dom0/swap

    • vgchange -a n qubes_dom0

    • cryptsetup luksClose /dev/mapper/luks-root

    • cryptsetup luksClose /dev/mapper/luks-*

    • wipefs -a /dev/nvme0n1

    • reboot

    BTRFS

    • swapoff /dev/mapper/luks-swap

    • cryptsetup luksClose /dev/mapper/luks-root

    • cryptsetup luksClose /dev/mapper/luks-swap

    • cryptsetup luksClose /dev/mapper/luks-*

    • wipefs -a /dev/nvme0n1p1

    • wipefs -a /dev/nvme0n1p2

    • reboot

Please remember that we do Detached Boot, so if you plan to use sys-usb (well you should), don’t forget to edit the kernel parameter in the grub menu if you want to update kernel, and add qubes_skip.autostart to prevent sys-usb starting and detaching our /boot partition.

13 Likes
2

Thanks for including a video of this. In many guides I think a video to follow along is sorely missing.

3

Just wanted to mention that I’ve been running swapless (didn’t even create a swap partition) for a while now and never had any issues (even though I use tens of vms at a time).

Did you mean to leave an iter time of 1?

Also, just out of curiosity, why did you chose twofish over aes to encrypt the boot partition?

4

Yea back then when my ram is only 16gb i didn’t even use swap, but if we look into the default installation, every distro do create swap.

ah my bad, it should be 10000, but since it write 1, it should be okay, as I’m sure one won’t skip my note here

others can too, it is preferred to cascade the ciphers as to not use the same one twice.

5

Please remember that we do Detached Boot, so if you plan to use sys-usb (well you should), don’t forget to edit the kernel parameter in the grub menu if you want to update kernel, and add qubes_skip.autostart to prevent sys-usb starting and detaching our /boot partition.

changes should be made to /etc/default/grub? I tried the following guide and successfully installed the OS but sys-usb is not installed. I want to use sys-usb i have ps2 keyboard and i want to use the latest kernel.

6

Have you tick box to create sys-usb in initial setup?
if you are not sure just create it.

  • If you use ps2 keyboard

qubesctl state.sls qvm.sys-usb

  • if not

qubesctl state.sls qvm.usb-keyboard

7

Doing that for the keyboard case, I got an error. (First off, it has to be done as sudo…second off, once I did that, the error message indicated it had no idea what qvm-usb-keyboard was.)

I just saw someone else give the same advice somewhere else and was about to make the same comment…but then I realized (s)he had typed qvm.usb-keyboard (a dot instead of a dash).

I’m not near qubeville right now (my personal name for the qubes box), so I can’t try it, but is that perchance what you meant? qvm dot sys-usb and qvm dot usb-keyboard?

8

my bad, it should be qvm.usb-keyboard.

2 Likes
9

I think you “swapped” the entries. nvme0n1p1 should be your luks-swap and nvme0n1p2 should be your root.

Isn’t this supposed to be mkswap /dev/mapper/luks-swap?

EDIT:

This should be grub2-mkconfig -o /boot/efi/EFI/qubes/grub.cfg

1 Like
10

Thanks for noticing, but i’m glad that the record is not swapped.

11

Amazing guide brother, truly useful and exactly what I was trying to accomplish.
I have almost completed the setup but I’m encountering an error after configuring dracut.
When using “swapoff /dev/mapper/luks-swap” I get the following error: “/sbin/swapoff: Input/output error”.
If you would have any idea what might be causing this issue, I would greatly appreciate it as I cannot find any information on this.

1 Like
12

Is the flash drive only needed at the time of booting the system? Can it be removed when the system is already running?

I would be grateful if you could help on a related issue :slight_smile:

13

Yes it is only used to decrypt the boot procedure.

1 Like
14

Thank you, that gave me clarity.

I correctly understand that the kernel parameter qubes_skip.autostart as written in the manual, will give the opportunity not to start sys-usb when decrypting, but when the system is already running, all USB controllers should go into the hands of sys-usb?

15

it depends on your threat model.

1 Like
16

I would like to install Qubes OS with a detached bootloader and detached LUKS header, specifically, with the booloader unencrypted. I do understand that this guide is for an encrypted bootloader, but I couldn’t find any guide that fits my goals, so this one must be close enough.

  1. Under the EXT4 instructions, you’ve put to run lvcreate -V30G -T qubes_dom0/root-pool -n root-pool, yet this results in this error: “Logical volume name root-pool and pool name must be different.” Should I just call this ‘root’?

  2. When running lvcreate -V30G -T qubes_dom0/root-pool -n root-pool and lvcreate -V800G -T qubes_dom0/vm-pool -n vm both commands result in four consistently displayed failure messages:

/usr/sbin/dmeventd: start failed: No such file or directory
WARNING: Failed to monitor qubes_dom0/root-pool
/usr/sbin/dmeventd: start failed: No such file or directory
WARNING: Failed to umonitor qubes_dom0/root-pool
/usr/sbin/dmeventd: start failed: No such file or directory
WARNING: Failed to monitor qubes_dom0/root-pool
/usr/sbin/dmeventd: start failed: No such file or directory
Logical volume “root” created.

  1. Your video seems to involve many additional steps that aren’t listed in the text of your guide, and anyone following the guide would only know after watching the more than half-hour long video.

a) You create two additional partitions on /dev/sdb that aren’t described in this guide.
b) The video doesn’t include additionally crucial steps for those of us building EXT4 based installations, such as properly configuring the hard disks under the Advanced Blivet GUI.
c) When creating the alias profiles, you don’t list `` in the text guide, in order to get the UUID for fstab.
d) You run shred -uvz /root-header and shred-uvz /swap-header yet there’s absolutely no references to these commands in the text.
e) While configuring grub, in your video you vi into the grub config and configure various settings. There’s no reference to these steps anywhere in your text.

  1. By the time of configuring grub, my grub config is empty except for GRUB_ENABLE_CRYPTODISK=y

This is where I stopped. Your kind help will be deeply appreciated, thank you.

17

Please tell me, I installed the test qubes for this, after adding the kernel parameter
qubes_skip.autostart
I run the command
grub2-mkconfig -o /boot/efi/EFI/qubes.cfg
I rebooted but nothing changes. my flash drive is connected to dom0

I decided to try the command
grub2-mkconfig -o /boot/grub2/grub.cfg
and it broke my test setup, why would this happen?

which command do i end up using?

thanks

18

Call it root-luks and rename the following instances too.

That’s alright. Shouldn’t prevent a successful execution.

That’s most likely due to running echo "GRUB_ENABLE_CRYPTODISK=y" > /etc/default/grub instead of echo "GRUB_ENABLE_CRYPTODISK=y" >> /etc/default/grub (notice the double >>. If you run it with a single > you’ overwrite the file, instead of appending at the end).

19

Thank you.

I’ve corrected the command to append instead of overwriting, but now when attempting to run grub2-mkconfig -o /boot/efi/EFI/qubes.cfg I am presented with this error:

/sbin/grub2-mkconfig: line 246: /boot/efi/EFI/qubes.cfg.new: No such file or directory'

I assume the original post is wrong, and the command should instead be grub2-mkconfig -o /boot/efi/EFI/grub.cfg.

This seems likely as in the video at 21:33, @51lieal is running the command for grub.cfg and not for qubes.cfg as they’ve written in their guide.

However, even when running the command for grub.cfg, the same error still results:

/sbin/grub2-mkconfig: line 246: /boot/efi/EFI/grub.cfg.new: No such file or directory'

20

Yes, change it to /boot/efi/EFI/qubes/grub.cfg