ZFS in Qubes OS

Use at your own risk!

Beware: Dragons might eat your precious data!

Install ZFS in Dom0

Install DKMS style packages for Fedora (defunct in 0.6.2 due to spl/issues/284)

Fetch and install repository for DKMS style packages for your Dom0 Fedora version http://zfsonlinux.org/fedora.html:

disp1# wget http://archive.zfsonlinux.org/fedora/zfs-release-1-1$(rpm -E %dist).noarch.rpm
dom0# qvm-run --pass-io disp1 'cat /home/user/zfs-release-1-1.fc18.noarch.rpm' > /home/user/zfs-release-1-1.fc18.noarch.rpm
dom0# sudo yum localinstall /home/user/zfs-release-1-1.fc18.noarch.rpm
dom0# sudo sed -i 's/$releasever/18/g' /etc/yum.repo.d/zfs.repo
dom0# sudo qubes-dom0-update @development-tools
dom0# sudo qubes-dom0-update zfs

Install DKMS style packages from git-repository

Build and install your DKMS or KMOD packages as described in http://zfsonlinux.org/generic-rpm.html.

Prerequisites steps in AppVM (i.e. disp1)

Checkout repositories for SPL and ZFS:

mkdir ~/repositories && cd ~/repositories
git clone https://github.com/zfsonlinux/spl.git
git clone https://github.com/zfsonlinux/zfs.git

Revert changes in SPL repository due to this bug: https://github.com/zfsonlinux/spl/issues/284

cd ~/repositories/spl
git config --global user.email "user@example.com"
git config --global user.name "user"
git revert e3c4d44886a8564e84aa697477b0e37211d634cd

Installation steps in Dom0

Copy repositories over to Dom0:

mkdir ~/repositories
qvm-run --pass-io disp1 'tar -cf - -C ~/repositories/ {spl,zfs}' | tar -xpf - -C ~/repositories/

Installing build requirements for SPL and ZFS DKMS modules:

sudo qubes-dom0-update dkms kernel-devel zlib-devel libuuid-devel libblkid-devel lsscsi bc autoconf automake binutils bison flex gcc gcc-c++ gdb gettext libtool make pkgconfig redhat-rpm-config rpm-build strace 

Configure and build SPL DKMS packages:

cd ~/repositories/spl
./autogen.sh
./configure --with-config=user
make rpm-utils rpm-dkms

Configure and build ZFS DKMS packages:

cd ~/repositories/zfs
./autogen.sh
./configure --with-config=user
make rpm-utils rpm-dkms

Install SPL and ZFS packages (i.e. version 0.6.2):

sudo yum localinstall \
    ~/repositories/spl/spl-0.6.2-1.qbs2.x86_64.rpm \
    ~/repositories/spl/spl-dkms-0.6.2-1.qbs2.noarch.rpm \
    ~/repositories/zfs/zfs-0.6.2-1.qbs2.x86_64.rpm \
    ~/repositories/zfs/zfs-dkms-0.6.2-1.qbs2.noarch.rpm \
    ~/repositories/zfs/zfs-dracut-0.6.2-1.qbs2.x86_64.rpm \
    ~/repositories/zfs/zfs-test-0.6.2-1.qbs2.x86_64.rpm

Configure ZFS

Automatically load modules

/etc/sysconfig/modules/zfs.modules

#!/bin/sh

for module in spl zfs; do
    modprobe ${module} >/dev/null 2>&1
done

Make this file executable.

Tuning

Tame the memory-eating dragon (i.e. 512 Mb zfs_arc_max):

/etc/modprobe.d/zfs.conf

options zfs zfs_arc_max=536870912

Setup a zpool with ZFS datasets

You can create a ZFS dataset for each AppVM, ServiceVM, HVM or TemplateVM or just use a pool as your backup location.

Move your existing directory to a temporary location, or the ZFS mount will overlay your directory.

Beware: VMs on a ZFS dataset aren’t working, if your ZFS installation deserts you.

So keep netvm, firewallvm and your templates on your root file-system (preferably on a SSD).

zpool create -m none -o ashift=12 -O atime=off -O compression=lz4 qubes mirror /dev/mapper/<cryptname1> /dev/mapper/<cryptname2>
zfs create -p qubes/appvms
zfs create -m /var/lib/qubes/backup-zfs qubes/backup
zfs create -m /var/lib/qubes/appvms/banking qubes/appvms/banking
zfs create -m /var/lib/qubes/appvms/personal qubes/appvms/personal
zfs create -m /var/lib/qubes/appvms/untrusted qubes/appvms/untrusted
zfs create -m /var/lib/qubes/appvms/work qubes/appvms/work

Have fun with zpool and zfs.

Tips and Hints

Backup your data

You’re depending on an huge amount of code for this file system, keep this in mind and backup your precious data.

Encrypt underlying devices

dom0# cryptsetup -c aes-xts-plain64 luksFormat <device1>
dom0# cryptsetup luksOpen <device1> <cryptname1>

With the use of cryptsetup a keyfile can be specified to decrypt devices.

dom0# head -c 256 /dev/random > /root/keyfile1
dom0# chmod 0400 /root/keyfile1
dom0# cryptsetup luksAddKey <device1> /root/keyfile1

Decrypt devices on boot

Add your devices to /etc/crypttab.

<cryptname1> <device1> <keyfile1>
<cryptname2> <device2> none

Specifying a keyfile is especially useful, if ZFS should be ready during boot.

Further Reading


This document was migrated from the qubes-community project
  • Page archive
  • First commit: 08 Dec 2020. Last commit: 08 Dec 2020.
  • Applicable Qubes OS releases based on commit dates and supported releases: 4.0
  • Original author(s) (GitHub usernames):
  • Original author(s) (forum usernames):
  • Document license: CC BY 4.0
1 Like

I gave obsolete information here that would confuse people

Quite the opposite: R4.2 now has a dedicated “zfs” Qubes OS storage driver, and the very latest ZFS filesystem prerelease version (2.2.0-rc3) added reflink support on Linux too so theoretically it should also be compatible with the “file-reflink” driver on both R4.1 and R4.2.

2 Likes

That’s great!

To confirm: “qubes/storage/zfs.py” does the VM creation/cloninig/management using ZFS, but the installer will still not have ZFS modules on it. So you’ll still need a non-zfs root, but once qubes is actually going, you can format the rest ZFS and use that.
Is that correct?

I think so.

Interesting development on this subject on qubes-public matrix channel thread, cross referencing to this post.

Basically, zfs-dkms is built with qubes-builder-v2, while not having landed yet under testing repos.

Repos:

@rustybird did you had a chance to check that out? Of course, we are not at the point of having zfs option from qubesos installer, but it seems logical to believe that since this advancement, it would be possible to go there more easily then adding beesd to qubesos iso and try to enhance BTRFS deployment which is not receiving love at all at Switch default pool from LVM to BTRFS-Reflink ¡ Issue #6476 ¡ QubesOS/qubes-issues ¡ GitHub

Long live ZFS!!! Can’t wait to test things out.

Also note that issue Create a benchmark suite for storage drivers ¡ Issue #7300 ¡ QubesOS/qubes-issues ¡ GitHub is progressing with tests: add simple storage performance tests by marmarek ¡ Pull Request #649 ¡ QubesOS/qubes-core-admin ¡ GitHub being the first step, so we can start comparing FS related changes, at least BTRFS/LVM soon, and ZFS once installer offers that option.

Oh! And prebuilt packages already exist:
https://yum.qubes-os.org/r4.2/current-testing/dom0/fc37/rpm/zfs-dkms-2.3.0-1.fc37.noarch.rpm

2 Likes

Thanks, great to see that these are now available. I’ve been curious about testing the ‘file-reflink’ storage driver on ZFS. (Probably not on my main system though, because sudo qubes-dom0-update zfs zfs-dkms wants to install 216 packages, waow)

A couple of tangentially-relevant things:

Putting zfs on a debian-12 template will require that that template actually uses a debian kernel (or a very old (6.1) fedora one).

It would be nice if you could select debian kernels from the menus that let you select which kernel a VM uses, just like you can with fedora kernels; instead I have to build the kernel into the template, and every template I clone from one like that has to copy that extra stuff.

For some reason if a VM with ZFS installed on it also has network access (via sys-firewall), it won’t work, as in the VM shuts down immediately on startup. I didn’t investigate too much; it could easily be something stupid I did. Or it could be an incompatibility between ZFS and the qubes-network code that has to be installed in a VM in order to connect to sys-firewall.