Install QUBES on NVMe ! (P15 gen2 - i7.11850 + 4x 32G RAM + 1x 512G NVMe + 2x 2TB NVMe)

Question about partitioning on 3 NVMe
My first NVMe is only 513GB and is used for heavy R/W, /tmp, swap, /VAR,
2nd NVMe 2To is for Qubes
3rd NVMe 2To is for Qubes
Q: How to I set up these two so as to have a single LVM ? is it risky in anyway to have an LVM spread over two disks ?
Or should I set it like:
NVMe1 = /root + common qubes (basic ones, Debian, Fedora)
NVMe2 = “Data” where I set extra qubes, like other distros

Then how ?

You can configure the disks to be in the same LVM or you can configure them as separate storages:

You can read how to configure the disks to be in the same LVM in some Fedora guides, it’ll be the same for Qubes OS installer.
No sure which way is better.

1 Like

Thks a lot for this !
What I’m concerned about, afraid about, is that … I break things. Allways did, and most likely allways will, that’s why I’ve always had separate partition (whichever OS) to keep my data, config and such (besides the usual swap, tmp, etc …)
So having everything inside a single LVM, is like … scary to me.
Someone here ssuggested about the imaging capability of dom0, so when you reinstall you can retrieve your previous set-up, but if everything is in the same LVM, that means a 4To backup (with all 6 or 8 qubes) and if one disk fail then the LVM will be broken and won’t run on a single disk (like RAID, right ?)
I know I’m tackling a way-bigger-than-me task here, and my fears are mostly because I lake the knowledge to fix things I break (unless obvious/easy) I usually just reinstall … and don’t want to reinstall every single qubes, besides the Fedora and Debian: Win10, Suse, Kali, Endless, etc …
Any suggestion to help my autistic brain seeing some light ?

right, so if I understood well, I could install Qubes on, say, a 200Gb partition on NVMe1, and then, once inside, I go through dom0 and create a (1800Gb-200Gb = 1600GB) pool and another 1800Pool on NVMe2, and have them joined to dom0,
So I would be able to create qubes on both, and they will remain if… (when, cause I’m sure I will) I have to reinstall Qubes on the first partition
Not sure if that would make 3 LVM, or one LVM with 3 partitions … I have to read further

Resume:
Install with /boot/efi /boot on a USB key didn’t work, but install with /boot/efi /boot on the main disk did work.

Installing Qubes now.

*USB key 2GB*
*SDC1 640MB /boot/efi FAT32*
*SDC2 1024MB /boot EXT4*
*SDC3 16MB /?? (for LUKS key)*
*SDC4 16MB /?? (for LUKS key)*

=-=-=-=-=-=
NVMe 1 256GB
NVMe2P1 64GB swap
NVMe2P2 32GB /tmp EXT2
NVMe2P3 32GB /var/tmp EXT2
NVMe2P1 110GB /var/log EXT2
=-=-=-=-=-=
NVMe 2+3 (RAID) 2TB
NVMe3P1 64GB / EXT4
NVMe3P2 99% /var/lib/qubes EXT4

Working as-is, no LVM, but now, I would like to find a solution for the USB key with boot, the idea is to have Qubes booting only if the key is in.

Take two: correction, /var/LIB/qubes (not: /var/qubes) :roll_eyes: :woman_facepalming:

USB key 2GB
SDC1 640MB /boot/efi FAT32
SDC2 1024MB /boot EXT4
SDC3 16MB /?? (for LUKS key)
SDC4 16MB /?? (for LUKS key)
=-=-=-=-=-=
NVMe 1 256GB
NVMe0n1p1 64GB swap
NVMe0n1p2 32GB /tmp EXT2
NVMe0n1p3 32GB /var/tmp EXT2
NVMe0n1p4 110GB /var/log EXT4
=-=-=-=-=-=
NVMe 2+3 (RAID) 2TB
NVMe1n1p1 128GB / EXT4
NVMe1n1p2 99% /var/lib/qubes EXT4
Contains:

  • /var/lib/qubes/vm-templates (templates directory; Debian=66GB; Fedora=72GB; Whonix=66GB, Kali=62GB,Win10=250GB)
  • /var/lib/qubes/appvms (“normal” qubes directory; Debian=6GB; Fedora=24GB; Vault=6GB; Whonix=6GB; default-dvm=6GB; contains also the sysVM (formerly /servicevms) sys-net=6GB; sys-usb=6GB; sys-vpn=66GB; sys-fw=4KB; sys-usb-proxy=6GB; etc …)
  • /var/lib/qubes/backups (stores all changes to the XML files, NOT the qubes-backup)
  • /var/lib/qubes/var/lib/qubes//updates (Dom0 updates?)
  • /var/lib/qubes/vm-kernels (The kernels that are available to use for the VMs)

=-=-=-=-=-=-=-=-=-=
USB HDD 8TB
/dev/sda1 backups; 6,7TB EXT4
/dev/sda2 reserved 512GB NTFS

And again … crashed the system, fried the 2GB USB key and lost the RAID1 2TB NVMe :confused:
I don’t even know what I’ve done this time …

Any idea why the installer would not see those two NVMe while they appear on the computer boot checkup and in Partition manager ?

Take three:

USB sdcard 32GB (I didn’t find smaller in my box)
SDC1 1GB /boot/efi FAT32
SDC2 2GB /boot EXT4
SDC3 32MB /?? (for LUKS key)
SDC4 32MB /?? (for LUKS key)
SDC5 32MB /?? (for LUKS key)
SDC6 99% /?? (reserved)
=-=-=-=-=-=
NVMe0n1 256GB (238.47GB)
LVM2 + LUKS vg_runtime
lv_Swap 64GB swap
lv_temp1 32GB /tmp EXT2
lv_temp2 32GB /var/tmp EXT2
lv_log 110GB /var/log EXT4
=-=-=-=-=-=
NVMe1n1+NVMe2n1 2x2TB (3752GB)
SOFT RAID1
Root 64GB / EXT4
luks_DATA 1799GB /var/lib/qubes EXT4

56min later, almost done (hang at “account.daemon.service”)
Tmrw: test the backup/restore, and if it work then I’ll be all set :slight_smile:

3 more install later …
Since using a USB stick for /boot, the sys-usb is not created, leading to all USB being connected to dom0, which is a pity, but according to this post, it makes sense as if the USB was attached to sys-usb it would not be able to boot the payload and would be accessible by any other USB, not better.

Let’s go for another install … without USB /boot
Unless crating a sys-usb after is still an option, while not giving it access to the boot USB (maybe attaching it permanently to dom0 ?)

aarrrr I don’T know, so many questions/options !

No matter how I partition and/or install (except for the full auto) I endup with this:

It stays there for ages.
If I brute-stop the computer and restart, I endup with a broken system, no sys-VM, no Q. drop-down menu

Partitions like:
NVMe0n1: nothing
NVMe1n1: ESP; Boot; 80GB /; 1780GB empty
NVMe2n1: empty; empty; 80GB / (mirror); 1780 empty
-=-=-=–=-=-=-=–=-=-
NVMe0n1: swap; temp; log
NVMe1n1: ESP; Boot; 80GB /; 96GB /var/lib/qubes; 1700GB empty
NVMe2n1: empty; empty; 80GB / (mirror); 96GB /var/lib/qubes (mirror); 1700GB empty
-=-=-=-=-=-=-=-=-=–=-
And many alternatives in-between

Any idea ? I’m out of options after about 20 tentatives

After 19 try/errors install of different variant, it will be:
No SDCard (to keep sys-usb)
No ZFS (not yet anyway)

NVMe 1 256GB
NVMe0n1p1 1GB /boot/efi
NVMe0n1p2 2GB /boot
NVMe0n1p3 64GB swap
NVMe0n1p4 32GB /tmp EXT2
NVMe0n1p5 32GB /var/tmp EXT2
NVMe0n1p5 80GB /var/log EXT4
NVMe0n1p6 19GB umount (resrved for ZFS one day)
=-=-=-=-=-=
NVMe 2+3 (RAID1) 2TB
NVMe1n1p1 80GB / EXT4
NVMe1n1p2 364GB /var/lib/qubes EXT4
NVMe1n1p3 1,6TB LVM2 umount (reserved for ZFS one day)
Contains:

  • /var/lib/qubes/vm-templates
  • /var/lib/qubes/appvms (“normal” qubes directory; Debian; Fedora; Vault; Whonix; default-dvm; sys-net; sys-usb; sys-vpn; sys-fw; sys-usb-proxy; etc …)
  • /var/lib/qubes/backups (changes to the XML files)
  • /var/lib/qubes/var/lib/qubes//updates
  • /var/lib/qubes/vm-kernels
    =-=-=-=-=-=-=-=-=-=
    USB HDD 8TB
    /dev/sda1 backups; 6,7TB EXT4 (attached to backup-vm)
    /dev/sda2 reserved 512GB NTFS

Update about the P15 Gen2:
1 NVMe 256GB for sysrun part (i.e swap) under the keyboard
2 NVMe 2TB for sysroot under the bottom hatch

  • BIOS RAID1 => No-go, Qubes doesn’t see the disks
  • ESP on SDcard => Can’t boot from SDcard reader, therefore SDcard on USB only (which prevent sys-usb)
  • Brightness => Keyboard buttons not usable, have to manually xrandr each boot
  • 3ple screen not managed, only 2 at the time (This is not specific to P15, this is an XFCE problem)
  • Thunder4 USB-c => Can’t boot (Win10) on USB if Thenderbolt is active
  • Intel UHD default
  • NVDIA AX2000 not used ? (need further investigation )

Did my HCL ! good girl :-p