Earlier this week, after discovering that 4.2.0rc5 simply won’t get along with Ventoy as an install launcher, I got what I thought was a workable configuration on an LSI 1.6TB PCIe flash drive. That being complete, I set about exploring how to make Qubes provide a similar level of storage security to what I do with Linux.
A small confession first - five years into using Redhat, way back in 2000, I became so disenchanted with the misery associated with RPM package management in those years, that I tore out everything Linux and replace them with a mix of FreeBSD and OpenBSD. I lightened up a bit by 2007 and let OpenSUSE on my desktop, but I didn’t really relent on server side stuff until I started working with the Java dependent Elasticsearch in late 2018.
I started into ZFS at the suggestion of a friend and with my history I had no trouble using FreeBSD Mastery: ZFS to get up to speed using it with Ubuntu. There was at the time a marvelous resource from Aaron Toponce on using ZFS on Linux, as smooth and well written as the FreeBSD book. These resources stand in sharp contrast to the endless muddle of material about BTRFS and LVM.
This short thread on the Proxmox subreddit gave voice to what I’d been suspecting about LVM.
LVM and similar techniques are a thing of the past, initially LVM didn’t support cache of any sort - I am talking about the first LVM ever in HP-UX. Linux adopted the LVM design from UNIX and slapped dm-cache on it however it shows that the initial LVM design was never done with caching in mind. ZFS was engineered from the ground up by SUN and for Solaris to have multiple levels and types of caching. It has significantly better caching algorithms and is safe in most configurations - except in the case of a single special device. All other types of caching in ZFS are meant to be destroyable without any risk of data loss. Even the SLOG can be lost and if your system doesn’t crash at precisely the same time your data will be committed to the pool itself from memory. Yes, you DON’T have to mirror the SLOG anymore but you might still want to so you can avoid performance degradation in case of SLOG failure.
Boomers and my fellow near-elderly Gen-X comrades will likely understand when I refer to LVM as a “bag on the side”. Every encounter I’ve ever had with it leaves me wondering what in the world the authors were thinking.
I think I first noticed BTRFS when I started touching Proxmox about nine months ago. Although the base install seems to require LVM, ZFS was available and I could simply import existing drives from the VirtualBox solution I was retiring. The history section on BTRFS provides comfort regarding the non-baggish nature of the solution.
In particular this paragraph:
In 2008, the principal developer of the ext3 and ext4 file systems, Theodore Ts’o, stated that although ext4 has improved features, it is not a major advance; it uses old technology and is a stop-gap. Ts’o said that Btrfs is the better direction because “it offers improvements in scalability, reliability, and ease of management”.[18] Btrfs also has “a number of the same design ideas that reiser3/4 had”.[19]
And this final line about it being the norm starting with Fedora 33.
In 2020, Btrfs was selected as the default file system for Fedora 33 for desktop variants.[34]
So, I already disliked and distrusted LVM based on occasional past encounters with it and after this recent research I know precisely why. BTRFS, on the other hand, seems to be a good idea that’s steadily maturing. I put a single SATA SSD and a pair of small PCIe NVMe carriers in my spare workstation and set about doing a Qubes 4.2.0rc5 install to compare support for BTRFS and ZFS.
I’d seen stuff here that I thought indicated ZFS was going to be included. It’s not part of the install process and I don’t see any sign of the needed packages post install. So much for that.
The BTRFS install, as it is in 4.2.0rc5, does NOT inspire confidence. I fed it the SATA SSD that had a previous LVM install on it and despite recovering all space, I was left with an install that only reclaimed three 20GB partitions, leaving no room for any templates, and resulting in a dysfunctional system. Removing the USB boot stick prior to reboot resulted in a spontaneous reboot that stalled with a single flashing underscore in the upper left hand corner of the screen.
A second attempt to install over the faulty BTRFS based install did work, but again removing the USB stick resulted in a spontaneous reboot and a wedge with the single underscore. Power cycling the machine did finally end with a functional system. While writing this I decided I’d have another go at a BTRFS install, which yielded a curious “Can’t exit boot loader” and a wedge. A power cycle of the system seems to have set it in motion again. The behavior of the Custom install option when selecting BTRFS is again non-deterministic. I took screen shots of the odd progression of partitions.
https://www.dropbox.com/scl/fo/reiav78wac36jzere0aqk/h?rlkey=nesg94ym3tixc8hlmglsytwtb&dl=0
My Unix experience having started sitting down in front of a DEC VT102 connected to a VAX running 4.2BSD thirty seven years ago, I’ve formed an opinion on Qubes.
This operating system is GLORIOUS. Wandering around the forums reminds of turn of the century OpenBSD, I keep expecting Theo to jump out and personally insult me for using too many polished words to express what are obviously faulty ideas 
I see so much innovation here, people working on problems I will never personally encounter for my use cases, but which after reading I can understand why they’re pursuing it. Qubes to Unix feels like Unix to hardware to me - the system re-imagined offers so much room to work on things I’d never even considered.
But … it would be great, for the sake of muggles who wander in here just wanting to do things online safely, if someone could write up a BTRFS, LVM & ZFS explainer. This needs to be calibrated for the previously Windows only twentysomething who’s trying to get their head around the concept that a single physical device can be more than one logical device.
I would even take a break from my “old man yelling at Proxmox cloud” duties to do this, but it would be great to hear from other documentation authors about how this is actually done in practice. And please don’t just send me back into the wilds of the forums