Hi all,
As I am sure I will have forgotten what I’ve done when the next version of Qubes rolls out, I wrote this howto for myself, and as a help for anyone who has similar ideas.
Goal: build a system that is redundant like raid, but that, unlike raid, doesn’t wear down all drives. In other words: have an unused hot-swap disk on standby, which doesn’t wear out as it is not used
My laptop has:
1 x M.2 2280 PCIe Gen4x4
1 x M.2 2280 PCIe Gen3x4
1 x 2.5″ SATA HDD support
I tried LVM raid, which worked well, until I removed one of the ssd’s as a test, then the laptop turns into a brick. So I opted for btrfs instead
What I did:
use a different liveUSB to partition every SSD, leaving 10% empty space between every partition for over-provisioning, so every partition can grow/shrink when needed. (efi 256Mb, boot 512Mb, rest luks with btrfs)
Useful commands to make the UUID more human readable:
mlabel -s -N ef1aaaaa :: -i '/dev/nvme0n1p1'
tune2fs -U aaaaaaaa-0000-0002-0000-aaaaaaaaaaaa /dev/nvme0n1p2
cryptsetup' with '--uuid=aaaaaaaa-0000-0003-0000-aaaaaaaaaaaa
mkfs.btrfs -f -L mylabel -U aaaaaaaa-bbbb-0003-bbbb-aaaaaaaaaaaa /dev/mapper/ssd0_crypt, or
btrfstune -U aaaaaaaa-bbbb-0003-bbbb-aaaaaaaaaaaa /dev/mapper/ssd0_crypt
create a subvolume for swap, with a swapfile as explained here Btrfs Swapfile - JWillikers
create subvolumes @qubes-4.1.2-dom0 and @qubes-4.1.2-varlibqubes and @qubes-4.2.0-dom0 and @qubes-4.2.0-varlibqubes (tested with 4.2.0-RC2 and 4.2.0 (leaving previous R4.1.2 as is))
make sure to chown and chmod @qubes-4.X-varlibqubes:
chown root:qubes @qubes-4.2.0-varlibqubes/
chmod 2770 @qubes-4.2.0-varlibqubes/
boot qubes installer, and when it asks where to install Qubes, select “advanced custom (blivet-gui)”
Format /boot and /boot/efi with the labels you want (remember to change the UUID later)
decrypt the luks container
select @qubes-4.X-dom0 as /
select @qubes-4.X-varlibqubes as /var/lib/qubes
=> if that fails, only set @qubes-4.X-dom0 as /
ignore swap, as Anaconda can’t handle swap-files (or I don’t know how to)
=> if anaconda won’t install qubes, then start over, reformat the top-level btrfs and install qubes in newly created subvolume or on the top level.
finish installation as normal
Make sure that after the install, and before the post-setup, to boot a liveUSB and change:
nano @qubes-4.2.0-dom0/root/anaconda-ks.cfg
and comment out all lines refering to btrfs subvolumes, like:
#btrfs none --noformat --useexisting btrfs.329
#btrfs /var/lib/qubes --subvol --name=@qubes-4.2.0-varlibqubes
#btrfs / --subvol --name=@qubes-4.2.0-dom0
If you forgot this step, and ended up with an empty qubes without any templates/appvms… then change /root/anaconda-ks.cfg now and rerun /usr/libexec/initial-setup/initial-setup-graphical manually
Remember to take plenty of snapshots along the way so you can roll back if something went wrong.
btrfs su sn -r @qubes-4.X-dom0 snapshots/@qubes-4.X-dom0-<date|time>-ro # (same for @qubes-4.X-<date|time>-varlibqubes)
[next step was needed for 4.2.0-RC2, but not needed for 4.2.0]
If mounting @qubes-4.X-varlibqubes on /var/lib/qubes during anaconda-install failed, then:
cd @qubes-4.2.0-dom0/var/lib/qubes/
cp -a --reflink=always * /path/to/\@qubes-4.2.0-varlibqubes/
rm -Rf *
and mount @qubes-4.2.0-varlibqubes on /var/lib/qubes in /etc/fstab
Go back to https://www.jwillikers.com/btrfs-swapfile and finish off what Anaconda couldn’t do automatically
to prevent unnecessary disk-writes (important for SSD’s), set ‘swappiness=1’
in dom0: nano /etc/sysctl.d/50-usersettings.conf # and add
vm.swappiness = 1
in debian (whonix) based templates: nano /etc/sysctl.conf# and add
vm.swappiness = 1
edit /etc/crypttab:
#└─nvme0n1p3 259:4 0 3.3T 0 part aaaaaaaa-0000-0003-0000-aaaaaaaaaaaa
luks-aaaaaaaa-0000-0003-0000-aaaaaaaaaaaa UUID=aaaaaaaa-0000-0003-0000-aaaaaaaaaaaa none
#└─nvme1n1p3 259:9 0 1.7T 0 part bbbbbbbb-0000-0003-0000-bbbbbbbbbbbb
luks-bbbbbbbb-0000-0003-0000-bbbbbbbbbbbb UUID=bbbbbbbb-0000-0003-0000-bbbbbbbbbbbb none
#└─sda3 259:0 0 6.6T 0 part cccccccc-0000-0003-0000-cccccccccccc
luks-cccccccc-0000-0003-0000-cccccccccccc UUID=cccccccc-0000-0003-0000-cccccccccccc none
edit /etc/fstab
#UUID=aaaaaaaa-bbbb-0003-bbbb-aaaaaaaaaaaa / btrfs defaults,x-systemd.device-timeout=0,discard,noatime,ssd,compress=zstd:3,space_cache,subvol=@qubes-4.1.2-dom0 0 0
#UUID=aaaaaaaa-bbbb-0003-bbbb-aaaaaaaaaaaa /var/lib/qubes btrfs defaults,x-systemd.device-timeout=0,discard,noatime,ssd,compress=zstd:3,space_cache,subvol=@qubes-4.1.2-varlibqubes 0 0
UUID=aaaaaaaa-bbbb-0003-bbbb-aaaaaaaaaaaa / btrfs defaults,x-systemd.device-timeout=0,discard,noatime,ssd,compress=zstd:3,space_cache,subvol=@qubes-4.2.1-dom0 0 0
UUID=aaaaaaaa-bbbb-0003-bbbb-aaaaaaaaaaaa /var/lib/qubes btrfs defaults,x-systemd.device-timeout=0,discard,noatime,ssd,compress=zstd:3,space_cache,subvol=@qubes-4.2.1-varlibqubes 0 0
#
#UUID=bbbbbbbb-bbbb-0003-bbbb-bbbbbbbbbbbb / btrfs defaults,x-systemd.device-timeout=0,discard,noatime,ssd,compress=zstd:3,space_cache,subvol=@qubes-4.2.1-dom0 0 0
#UUID=bbbbbbbb-bbbb-0003-bbbb-bbbbbbbbbbbb /var/lib/qubes btrfs defaults,x-systemd.device-timeout=0,discard,noatime,ssd,compress=zstd:3,space_cache,subvol=@qubes-4.2.1-varlibqubes 0 0
#
#UUID=cccccccc-bbbb-0003-bbbb-cccccccccccc / btrfs defaults,x-systemd.device-timeout=0,discard,noatime,ssd,compress=zstd:3,space_cache,subvol=@qubes-4.2.1-dom0 0 0
#UUID=cccccccc-bbbb-0003-bbbb-cccccccccccc /var/lib/qube btrfs defaults,x-systemd.device-timeout=0,discard,noatime,ssd,compress=zstd:3,space_cache,subvol=@qubes-4.2.1-varlibqubes 0 0
#
UUID=aaaaaaaa-bbbb-0003-bbbb-aaaaaaaaaaaa /swap btrfs defaults,x-systemd.device-timeout=0,discard,noatime,ssd,space_cache,subvol=@swap 0 0
#UUID=bbbbbbbb-bbbb-0003-bbbb-bbbbbbbbbbbb /swap btrfs defaults,x-systemd.device-timeout=0,discard,noatime,ssd,space_cache,subvol=@swap 0 0
#UUID=cccccccc-bbbb-0003-bbbb-cccccccccccc /swap btrfs defaults,x-systemd.device-timeout=0,discard,noatime,ssd,space_cache,subvol=@swap 0 0
/swap/swapfile none swap defaults 0 0
#
UUID=aaaaaaaa-bbbb-0002-bbbb-aaaaaaaaaaaa /media/nvme0-boot ext2 defaults 1 2
UUID=EF1A-AAAA /media/nvme0-boot/efi vfat umask=0077,shortname=winnt,discard 0 2
#UUID=bbbbbbbb-bbbb-0002-bbbb-bbbbbbbbbbbb /media/nvme1-boot ext2 defaults 1 2
#UUID=EF1B-BBBB /media/nvme1-boot/efi vfat umask=0077,shortname=winnt,discard 0 2
#UUID=cccccccc-bbbb-0002-bbbb-cccccccccccc /media/sda-boot ext2 defaults 1 2
#UUID=EF1C-CCCC /media/sda-boot/efi vfat umask=0077,shortname=winnt,discard 0 2
#
# /media/nvme0-boot /boot none defaults,bind 0 0
# /media/nvme0-boot/efi /boot/efi none defaults,bind 0 0
#
UUID=aaaaaaaa-bbbb-0003-bbbb-aaaaaaaaaaaa /media/aaaaaaaa-bbbb-0003-bbbb-aaaaaaaaaaaa btrfs defaults,x-systemd.device-timeout=0,discard,noatime,ssd,compress=zstd:3,space_cache 0 0
UUID=bbbbbbbb-bbbb-0003-bbbb-bbbbbbbbbbbb /media/bbbbbbbb-bbbb-0003-bbbb-bbbbbbbbbbbb btrfs defaults,x-systemd.device-timeout=0,discard,noatime,ssd,compress=zstd:3,space_cache 0 0
UUID=cccccccc-bbbb-0003-bbbb-cccccccccccc /media/cccccccc-bbbb-0003-bbbb-cccccccccccc btrfs defaults,x-systemd.device-timeout=0,discard,noatime,ssd,compress=zstd:3,space_cache 0 0
after changing /etc/fstab, run findmnt --verify --verbose
edit /etc/default/grub
GRUB_CMDLINE_LINUX="rd.luks.uuid=luks-aaaaaaaa-0000-0003-0000-aaaaaaaaaaaa plymouth.ignore-serial-consoles 6.6.2-1.qubes.fc37.x86_64 x86_64 rhgb quiet"
#GRUB_CMDLINE_LINUX="rd.luks.uuid=luks-bbbbbbbb-0000-0033-0000-bbbbbbbbbbbb plymouth.ignore-serial-consoles 6.6.2-1.qubes.fc37.x86_64 x86_64 rhgb quiet"
#GRUB_CMDLINE_LINUX="rd.luks.uuid=luks-cccccccc-0000-0033-0000-cccccccccccc plymouth.ignore-serial-consoles 6.6.2-1.qubes.fc37.x86_64 x86_64 rhgb quiet"
Make sure to make plenty of snapshots along the way. And when dom0 is updated and customized as you want, send your final snapshots to the other disks:
btrfs su sn -r @qubes4.1.2-dom0 snapshots/@qubes4.1.2-dom0-<date/version>-ro
btrfs send snapshots/@qubes4.1.2-dom0-<date/version>-ro | btrfs receive /path/to/other/SSDs # `https://wiki.archlinux.org/title/Btrfs#Send/receive`
btrfs su sn /path/to/other/SSD/snapshots/@qubes4.1.2-dom0-<date/version>-ro /path/to/other/SSD/@qubes4.1.2-dom0
After moving/copying qubes to a new disk, or after changing the uuid, or subvolume-name, make sure to update the correct uuid/subvol in:
/etc/fstab/boot/efi/EFI/qubes/grub.cfg=> the line ‘search --no-floppy --fs-uuid --set=dev XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX’/etc/default/grub=> the line ‘GRUB_CMDLINE_LINUX=“rd.luks.uuid=luks-XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX”’/etc/grub.d/40_customand rerungrub2-mkconfig -o /boot/grub2.grub.cfg
test setup by booting in every dom0 on every disk.
physically remove one or more disks and test if laptop still boots when one or more disks fails. (make sure to comment out the references to the removd disk in '/etc/fstab)
Keep in mind: this approach does not replace backups!
compress=zstd:3 turns dom0 from 4Gb to 2Gb
compsize original installed 4.2.0
Processed 176186 files, 82608 regular extents (88510 refs), 73123 inline.
Type Perc Disk Usage Uncompressed Referenced
TOTAL 79% 4.0G 5.1G 5.7G
none 100% 3.7G 3.7G 4.2G
zstd 23% 341M 1.4G 1.4G
prealloc 100% 28K 28K 16M
compsize same 4.2.0 transferred to other SSD and after upgrading to 4.2.1
Processed 176178 files, 88613 regular extents (96716 refs), 83239 inline.
Type Perc Disk Usage Uncompressed Referenced
TOTAL 40% 2.0G 5.1G 5.7G
none 100% 781M 781M 816M
zstd 30% 1.3G 4.3G 4.9G
Problems I wasn’t able to solve:
Template with btrfs as root filesystem: I tried @51lieal suggestion of Btrfs for template/appvm - #2 by 51lieal but I was unable to make it work. It’s simply too complicated for my simple mind :-s. Would anyone be willing to create a, preferably, debian-12-xfce template and distribute it through one of the official qubes repositories?
Alternatively…
mounting a @subvolume inside a VM. => when, for example, cccccccc-bbbb-0003-bbbb-cccccccccccc is not mounted in dom0, then I am able to mount the whole disk (top-level) into any VM. But I am unable to mount a @subvolume of a, in dom0 mounted, btrfs partition.
Creating an image file, formatting as btrfs and mounting inside a VM works. But I don’t want to allocate disk-space to a file… I would like to mount a specific subvolume inside an appVM. Any tips on how to do this?
In any case, after playing around with LVM and BTRFS for a while, in my opinion, BTRFS is easier to manage, and not having to allocate disk space up front, is a huge advantage. E.g. on my old laptop, one time I didn’t have enough room on my root-pool which can be increased, if there’s enough disk space left. But since I couldn’t decrease the size of the vm-pool…, I ended up having to re-install qubes, restoring all the VM’s from backup, which is a problem that doesn’t exist with BTRFS.
Does anyone know in which system file the appVM’s UUID are stored? That would be handy in case one has to swap over one varlibqube-subvolume to a different dom0-subvolume
If I made any errors, or there are better ways to achieve what I did, please let us know.
Thanks!
Hopefully this is useful to someone.