Install /boot on a separate medium

Erica.vH · April 23, 2023, 8:14pm

Hi,
As Qubes page “Custom installation | Qubes OS” is outdated, I’m navigating blind here.
I’v DL’ed 4.1.2 and install it clean.
I want my /boot/efi and /boot on a USB stick
I have 4 drives,
32Gb sdb with install 4.1.2
2GB sda usb for Qubes boot (computer start without usb stick goes straight to vanilla Linux Suse)
512GB NVMe0 with Linux Suse and Win
2TB Nvme1 for Qubes
2TB NVMe2 for Qubes

What I want:
sda1 /boot/efi
sda2 /boot
NVMe0p13 32GB /tmp (Covered by /inside Lucks)
NVMe0p14 32GB swap (Covered by /inside Lucks)

NVMe1+NVMe2 90% LVM Qubes with 512GB /home
-=-=-=-=-=-=-==-=-=
Qubes install has 3 options:
Automatic, Manual Cutom, manuel custom blivet-GUI
Obviously, I can’t use the Auto
But neither the custom nor custom-Bliviet allow me to do this simple set-up

In custom, I’ve added the two boot, the swap, the tmp, but I can’t get them inside the Lucks, and then when I want to set / + /home on the joint-2 NVMe, it crash
in custom-Bliviet, I can set an LVM to join the 2 NVMe but can’t set which part for / and which part for /home (I’d like to have /home separated so as to not lose it everytime I reinstall) and again, can’t get the swap and /tmp inside the lucks

Anyone to guide me here ? pretty pleeeeease

Erica.vH · April 23, 2023, 8:17pm

The easest would be to just go with LVM on NVMe1+NVMe2 and the /boot on sda
But then I won’t have the NVMe0p13 (/tmp) + NVMe0p14 (/swap) inside the LUKS

Erica.vH · April 23, 2023, 10:07pm

The current “Redirecting…” is for a single drive full install.
Here is my adaptation of it as I interpret it:
-=-=-=-=-=-=-=-=-=-=-=-=—=-

Example: Custom LUKS Configuration

Boot into the Qubes installer, then press ctrl+alt+F2 to get a virtual console.

(Optional) Wipe the disk:

 # dd if=/dev/zero of=/dev/sda bs=1M status=progress && sync

Create partitions:

 # fdisk /dev/sda
 # fdisk /dev/NVMe1
 # fdisk /dev/NVMe2

Follow the steps to create (two => SIX) partitions:

sda1 ~1GiB for /boot/efi (not included in LUKS)
sda2 ~1GiB for /boot (included or not included in LUKS)
NVMe0p13 ~32GiB for /tmp (included in LUKS)
NVMe0p14 ~32GiB for /swap (included in LUKS)
NVMe1p1 ~1,8TB for / (included in LUKS)
NVMe2p1 ~1,3TB for / (included in LUKS)
NVMe2p2 ~0.5TB for /home (included in LUKS)

Create LUKS encrypted volume:

 # cryptsetup -v -type luks2 --hash sha512 --cipher argon2id --key-size 512 --use-random --iter-time 10000 --verify-passphrase luksFormat /dev/sda2

==> How do I go with my 4 partition ?? one by one ? or do I create an LV first and then create partition within ? everthing following is untouched, as this is where I’m stuck.

-=-=-===–=-=-=-=-=-=-=—=-=
4. Open encrypted volume:

 # cryptsetup open /dev/sda2 luks

Create LVM volumes:

 # pvcreate /dev/mapper/luks
 # vgcreate qubes_dom0 /dev/mapper/luks
 # lvcreate -n swap -L 10G qubes_dom0
 # lvcreate -T -l +100%FREE qubes_dom0/pool00
 # lvcreate -V1G -T qubes_dom0/pool00 -n root
 # lvextend -L <size_of_pool00> /dev/qubes_dom0/root

Proceed with the installer. You can do that either by pressing ctrl+alt+F6, or by rebooting and restarting the installation. At the disk selection screen, select:

 [x] I will configure partitioning.
 [ ] Encrypt my data.

Decrypt your partition.

augsch · April 24, 2023, 2:09am

I’d like to point to detached header, in case you are not aware of this guide. The instructions include moving boot and efi partitions to a usb drive.

Erica.vH · April 24, 2023, 9:12pm

Moving it later to the usb rather than direct it to the usb at install ?
I take note, but then what will hapen to the partition created for booton the main disk during install ? Will I be able to merge it with the / partition (inslide LUKS)

Erica.vH · April 29, 2023, 12:13pm

Oh ! I think I get what I’m doing wrong …

I’m confusing creating a partition and formatting a partition.

So in 2) create partitions I should NOT have any: " /xyz "

Erica.vH:

2. Create partitions:
#fdisk /dev/sda
– sda1 ~1GiB for ‘/boot/efi’ => NOT formatted
– sda2 ~1GiB for /boot => NOT formatted

#fdisk /dev/NVMe0
– NVMe0p13 ~32GiB for /tmp => NOT formatted
– NVMe0p14 ~32GiB for /swap => NOT formatted

#fdisk /dev/NVMe1

NVMe1p1 ~1,8TB for / => NOT formatted

#fdisk /dev/NVMe2

NVMe2p1 ~1,3TB for / => NOT formatted

NVMe2p2 ~0.5TB for /home => NOT formatted

NOW, for the fun part … LUKS first or LVM first ?
=> If I follow the guide, I’m supposed to :
Create LUKS encrypted volume before the LVM, But that would mean I will end-up with SIX LUKS entry, while if I invert 3) and 5) meaning I create LVM before LUKS, I will have my 6 partition inside an LVM, and therefore only one LUKS entry

* # Example adapted from the guide:
3. Create LUKS encrypted volume:

- #cryptsetup -v -type luks2 --hash sha512 --cipher argon2id --key-size 512 --use-random --iter-time 11111 --verify-passphrase luksFormat /dev/sda2
- #cryptsetup -v -type luks2 --hash sha512 --cipher argon2id --key-size 512 --use-random --iter-time 11111 --verify-passphrase luksFormat /dev/NVMe0p13
- #cryptsetup -v -type luks2 --hash sha512 --cipher argon2id --key-size 512 --use-random --iter-time 11111 --verify-passphrase luksFormat /dev/NVMe0p14
- #cryptsetup -v -type luks2 --hash sha512 --cipher argon2id --key-size 512 --use-random --iter-time 11111 --verify-passphrase luksFormat /dev/NVMe1p1
- #cryptsetup -v -type luks2 --hash sha512 --cipher argon2id --key-size 512 --use-random --iter-time 11111 --verify-passphrase luksFormat /dev/NVMe2p1
- #cryptsetup -v -type luks2 --hash sha512 --cipher argon2id --key-size 512 --use-random --iter-time 11111 --verify-passphrase luksFormat /dev/NVMe2p2

Open encrypted volume:

- #cryptsetup open /dev/sda2 luks
- #cryptsetup open /dev/NVMe0p13 luks
- #cryptsetup open /dev/NVMe0p14 luks
- #cryptsetup open /dev/NVMe1p1 luks
- #cryptsetup open /dev/NVMe2p1 luks
- #cryptsetup open /dev/NVMe2p2 luks

Create LVM volumes:

 -# pvcreate /dev/mapper/luks
 -# vgcreate qubes_dom0 /dev/mapper/luks
 -# lvcreate -n swap -L 32G qubes_dom0 (I want it on NVMe0p13)
 -# lvcreate -n /tmp -L 32G qubes_dom0 (I want it on NVMe0p13)
 -# lvcreate -T -l +90%FREE qubes_dom0/pool00
 -# lvcreate -V1G -T qubes_dom0/pool00 -n root
 -# lvextend -L <size_of_pool00> /dev/qubes_dom0/root
 -# lvextend -L <size_of_pool00> /dev/qubes_dom0/home (I want it on NVMe2p2)

And there again, how would I make sure the swap and the /tmp are on the NVMe0 disk, not on the NVMe1 nor NVMe2 (NVMe0 is smaller = cheaper so I want to redirect heavy R/W to this one) and that the /home is on NVMe2p2 (so I can keep it safe away for reformatting for reinstalling)

cayce · April 29, 2023, 3:22pm

I could be wrong but, AFAIK you are not going to be able to use this cipher for your boot volume with r4.1.2 (only luks1 is supported in the shipping grub2 version).

Remove the cipher flag & change your type flag to luks1.

Alternatively, transition to r4.2.

YMMV

renehoj · April 29, 2023, 3:46pm

Can’t you just do a normal install and use luksCovertKey to change the KDF from argon2i to argon2id?

Erica.vH · April 29, 2023, 4:04pm

Yes, I sure can, but that is secondary.
For now, I’M not focused on this 2nd chapter …
For now I’m focusing on “actually installing” Qubes according to my partition wish

Erica.vH · April 29, 2023, 4:09pm

You are not wrong, /boot/efi is outside the LUKS,
That’s why in my partitioning wish I keep it out …

sda1 (which will be /boot/efi) will NOT be included in LUKS
sda2 (which will be /boot) will be included
NVMe0p13 (which will be /swap) will be included
NVMe1p1 (which will be /) will be included
NVMe2p1 (which will be /) will be included
NVMe2p2 (which will be /home) will be included
But I just can’t past this part … and wait for someone to guide me through

cayce · April 29, 2023, 4:14pm

Looks like you are trying to use luks2 and associated cipher for /dev/sda2 (which you’re planning to use for /boot). Current r4.1 shipping grub2 = 2.04.

Again, what you’ve plotted will not result in a booting system.

Erica.vH · April 29, 2023, 4:15pm

aaarrrggg mixing things up again …

Partition to be created,
Partition to be formatted,
Partition NOT to be attributed a mount point.

[quote=“Erica.vH, post:6, topic:18188, full:true”]
Oh ! I think I get what I’m doing wrong …
I’m confusing creating a partition and formatting a partition.

Erica.vH:

2. Create partitions:
#fdisk /dev/sda
– sda1 ~1GiB for ‘/boot/efi’ => **Formatted (XFS, EXT3, EXT4,) ** but NOT mounted
– sda2 ~1GiB for /boot => idem

#fdisk /dev/NVMe0
– NVMe0p13 ~32GiB for /tmp => idem

etc, etc …

So … I sure can create partition, I certainly can format it,
But I can’t figure out how to get them all inside LUKS and LVM, while forcing some (swap, tmp, boot, home, …) to be on specific drive/partition.

Erica.vH · April 29, 2023, 4:22pm

As both sda1 /boot/efi (not crypted) and sda2 /boot (potentially crypted) are on a USB stick, there is no problem for me to keep /boot in the clear

cayce · April 29, 2023, 4:24pm

Then why are you doing so with:

cryptsetup -v -type luks2 --hash sha512 --cipher argon2id --key-size 512 --use-random --iter-time 11111 --verify-passphrase luksFormat /dev/sda2

Erica.vH · April 29, 2023, 4:45pm

Because when I do a full auto install (on one disk only) I see that /boot/efi is out of LUKS while /boot is inside LUKS, as discussed further up in this thread.

cayce · April 29, 2023, 4:48pm

You’re a bit confused about “LUKS”.

Can’t say I didn’t try …

cayce · April 29, 2023, 5:01pm

I “think” this may be what you’re after …

# @erica's post-partitioning pre-install preparation
cryptsetup -v -type luks2 --hash sha512 --cipher argon2id --key-size 512 --use-random --iter-time 11111 --verify-passphrase luksFormat /dev/nvme0n1p1
cryptsetup open /dev/nvme0n1p1 luks-root
pvcreate /dev/mapper/luks-root
vgcreate qubes_dom0 /dev/mapper/luks-root
lvcreate -T -L 1800G qubes_dom0/root-pool

cryptsetup -v -type luks2 --hash sha512 --cipher argon2id --key-size 512 --use-random --iter-time 11111 --verify-passphrase luksFormat /dev/nvme0n2p1
cryptsetup open /dev/nvme0n2p1 luks-idk
pvcreate /dev/mapper/luks-idk
vgcreate qubes_dom0 /dev/mapper/luks-idk
lvcreate -T -L 1300G qubes_dom0/idk-pool

cryptsetup -v -type luks2 --hash sha512 --cipher argon2id --key-size 512 --use-random --iter-time 11111 --verify-passphrase luksFormat /dev/nvme0n2p2
cryptsetup open /dev/nvme0n2p2 luks-home
pvcreate /dev/mapper/luks-home
vgcreate qubes_dom0 /dev/mapper/luks-home
lvcreate -T -L 500G qubes_dom0/home-pool

cryptsetup -v -type luks2 --hash sha512 --cipher argon2id --key-size 512 --use-random --iter-time 11111 --verify-passphrase luksFormat /dev/nvme0n1p13
cryptsetup open /dev/nvme0n1p13 luks-tmp
pvcreate /dev/mapper/luks-tmp
vgcreate qubes_dom0 /dev/mapper/luks-tmp
lvcreate -T -L 32G qubes_dom0/tmp-pool

cryptsetup -v -type luks2 --hash sha512 --cipher argon2id --key-size 512 --use-random --iter-time 11111 --verify-passphrase luksFormat /dev/nvme0n1p14
cryptsetup open /dev/nvme0n1p14 luks-swap
pvcreate /dev/mapper/luks-swap
vgcreate qubes_dom0 /dev/mapper/luks-swap
lvcreate -T -L 32G qubes_dom0/swap-pool

Then go back to the calamares installer and define/format/assign mount points there.

YMMV

It is unclear to me what/where /dev/nvme0n2p1 is/will be mounted thus, I’ve assigned it “IDK”.

Erica.vH · April 29, 2023, 5:59pm

Looks pretty much like what I have in mind indeed !
As you have pointed out, I’m confused about LUKS (amongst many other things ! hihi)

Will this set up leads to unlocking 6 times at boot-up ? or will them all be seen as a single luks to be unlocked once ?

cayce · April 29, 2023, 6:03pm

Not quite at boot but, at mount. This would be quite the task for you to auto-magically sort out.

I in no way suggest proceeding as you’ve outlined.

Definitely NOT.

Erica.vH · April 29, 2023, 6:03pm

cayce:

I “think” this may be what you’re after …
# @erica's post-partitioning pre-install preparation
YMMV

It is unclear to me what/where /dev/nvme0n2p1 is/will be mounted thus, I’ve assigned it “IDK”.

I have 4 drives,

USB stick for boot
NVMe0 with 15 partitions, p13 and p14 for Qubes
NVMe1 with 1 partition for Qubes root
NVMe2 with 2 partition, p1 for the rest of Qubes root; and p2 for /home

Not sure where that p1 on NVMe0 comes from either, as on NVMe0 the p1 is /boot/efi for SUSE