BTRFS and Qubes OS

Insurgo · November 24, 2024, 3:30pm

Switch default pool from LVM to BTRFS-Reflink

opened 09:46AM - 22 Mar 21 UTC

T: enhancement P: default C: storage

**The problem you're addressing (if any)** In R4.0, the default install uses …LVM thin pools. However, LVM appears to be optimized for servers, which results in several shortcomings: - Space exhaustion is handled poorly, requiring manual recovery. This recovery may sometimes fail. - It is not possible to shrink a thin pool. - Thin pools slow down system startup and shutdown. Additionally, LVM thin pools do not support checksums. This can be achieved via dm-integrity, but that does not support TRIM. **Describe the solution you'd like** I propose that R4.3 use BTRFS+reflinks by default. This is a proposal ― it is by no means finalized. **Where is the value to a user, and who might that user be?** BTRFS has checksums by default, and has full support for TRIM. It is also possible to shrink a BTRFS pool without a full backup+restore. BTRFS does not slow down system startup and shutdown, and does not corrupt data if metadata space is exhausted. When combined with LUKS, BTRFS checksumming provides authentication: it is not possible to tamper with the on-disk data (except by rolling back to a previous version) without invalidating the checksum. Therefore, this is a first step towards untrusted storage domains. Furthermore, BTRFS is the default in Fedora 33 and openSUSE. Finally, with BTRFS, VM images are just ordinary disk files, and the storage pool the same as the dom0 filesystem. This means that issues like #6297 are impossible. **Describe alternatives you've considered** None that are currently practical. bcachefs and ZFS are long-term potential alternatives, but the latter would need to be distributed as source and the former is not production-ready yet. **Additional context** I have had to recover manually from LVM thin pool problems (failure to activate, IIRC) on more than one occasion. Additionally, the only supported interface to LVM is the CLI, which is rather clumsy. The LVM pool requires nearly twice the amount of code as the BTRFS pool, for example. **Relevant [documentation](https://www.qubes-os.org/doc/) you've consulted** `man lvm` **Related, [non-duplicate](https://www.qubes-os.org/doc/reporting-bugs/#new-issues-should-not-be-duplicates-of-existing-issues) issues** #5053 #6297 #6184 #3244 (really a kernel bug) #5826 #3230 ― since reflink files are ordinary disk files we could just rename them without needing a copy #3964 everything in https://github.com/QubesOS/qubes-issues/search?q=lvm+thin+pool&state=open&type=issues Most recent benchmarks: https://github.com/QubesOS/qubes-issues/issues/6476#issuecomment-1689640103

@rustybird said

the only thing I’m aware of that needs some tweaking in the installer and in qubes-dist-upgrade is not to hardcode discard[=sync] in fstab.)

Of course, that applies to dom0 fstab as well.

Also, this issue Working plan to have better qubesos btrfs settings from install · Issue #211 · tasket/wyng-backup · GitHub
(I gave up testing bees for now. Wyng/beesd would need /var/lib/qubes to be subvolume of rootfs, and dom0 would need to be either sepetate rootfs volume or root/private disk images as all other qubes to ease dom0 backup and restore…)

Feel free to comment in issues torward better implementation and installer defaults.