ZFS tuning/usage on Qubes OS

[disclose-mushy]

ZFS users, do you tune ZFS for your system?

• Do you use dedup for your VMs?
• What recordsize do you use?
• What compression algo do you use?
• Which Qubes pool driver do you use, ZFS or file-reflink?
• How do you organize your datasets?
• How do you mitigate/handle zpool fragmentation?
• Do you use ZFS as your rootFS?

I’m mainly asking because I struggle with zpool fragmentation, but I’m also wondering how other users setup their systems.

[Euwiiwueir]

Hi fellow ZFS enthusiast,

@disclose-mushy

• Do you use dedup for your VMs?

No. Have wanted to try this, but haven’t done the experiment:

@Euwiiwueir

I am curious if it could have useful application specifically for a TemplateVM dataset in the zpool, for users who have many custom templates. I have very many templates, most of which are dupes of each other modulo a few packages. If all of these templates, and only templates, and perhaps only the root images of the templates, were placed in a dataset with deduplication enabled and a large recordsize (1 MB or greater) then perhaps the costs would be workable.

---

• What recordsize do you use?

See:

@likeafox

recordsize isn’t applicable to qubes, because qubes use zfs volumes not filesystems. recordsize is for filesystems. I think the analog for volumes would be volblocksize.

---

• What compression algo do you use?

I have been using the default lz4, but @likeafox makes a compelling point IMO:

@likeafox

I won’t say it could never be an acceptable risk, however I’m never going to recommend anyone use compression on a Qubes pool, because it breaks isolation between dom0 and your qubes. By using compression you are tasking dom0 with running compression/decompression algorithms on data that is provided by guests. Those are algorithms which are not immune to security vulnerabilities. In fact I would recommend anyone currently using it to disable compression at least until they’ve seriously considered the risk.

---

• Which Qubes pool driver do you use, ZFS or file-reflink?

ZFS

• How do you organize your datasets?

Currently, one dataset for dom0 root, one for dom0 swap (possibly risky but has never been an issue), and one dataset for a single vm-pool. I would like to try an additional dataset for template dedup (referred to above) but I have enough free space there hasn’t been a need for it beyond curiosity.

• How do you mitigate/handle zpool fragmentation?

Can you say more about what you’re observing?

• Do you use ZFS as your rootFS?

Yes; has been working fine, and it makes me feel good, but it’s a little dicey to set up.

[disclose-mushy]

@Euwiiwueir

See:

recordsize isn’t applicable to qubes, because qubes use zfs volumes not filesystems. recordsize is for filesystems. I think the analog for volumes would be volblocksize.

Sorry for the confusion. I use file-reflink because ZFS driver leaks VM names onto disk, so recordsize is applicable in that case.

@Euwiiwueir

I have been using the default lz4, but @likeafox makes a compelling point IMO:

Wouldn’t that also apply to the official Qubes OS backups tool, as it allows backups to be compressed? Maybe the storage domain could fix this, idk.

@Euwiiwueir

Can you say more about what you’re observing?

I have accumulated about 40% Fragmentation over 2 months of usage at 128KiB recordsize.

[rustybird]

@disclose-mushy

I use file-reflink

Interesting! You’re the first user of file-reflink on ZFS I know of.

@disclose-mushy

ZFS driver leaks VM names onto disk

That would be with ZFS’s native encryption then, instead of LUKS?

@disclose-mushy

I have accumulated about 40% Fragmentation over 2 months of usage at 128KiB recordsize.

Have you experienced performance getting worse? Like for VM startup/shutdown/cloning/removal, or in-VM I/O?

AFAIU zpool fragmentation measures how fragmented the free space is, rather than the files: https://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSZpoolFragmentationMeaning

Try running sudo filefrag on your VMs’ private.img files (and also root.img for TemplateVMs and StandaloneVMs) if you’re curious about their fragmentation. Assuming that this utility works on ZFS. (On Btrfs and XFS too IIRC, fragmentation starts being noticeable in the millions-of-extents range during VM startup/shutdown/etc.)

[likeafox]

• no dedup
• default recordsize/volblocksize
• gzip for a few qubes, no compression for most
• zfs qubes pool driver
• I don’t worry about fragmentation

[root@dom0 ~]# zfs list -H -d 1 -o name,reserv mainpool
mainpool        none
mainpool/dom0   25G
mainpool/extras none
mainpool/qubes  none

Disk layout is just a ZFS vdev inside a LUKS container. 25G reserved to dom0. “extras” dataset contains volumes that I attach to running qubes as needed.

---

@disclose-mushy

Wouldn’t that also apply to the official Qubes OS backups tool, as it allows backups to be compressed? Maybe the storage domain could fix this, idk.

Correct. The backup tool’s compression option should not be used, just in my opinion.

[rustybird]

Re: the Qubes OS backup system doing compression of potentially malicious VMs’ data, I’d take that the other way around: It’s an indication that this isn’t considered a security risk, because the backup system is almost ruthlessly oriented towards security above other concerns. (E.g. people have repeatedly asked for an option disabling encryption to make incremental backups feasible, but this has been rejected because the current implementation ties encryption to end-to-end integrity protection.) If compressing VM data in dom0 was judged to be a security risk, it would probably not be available at all in the backup system, let alone enabled by default…

(OTOH I could see choosing a tried-and-true compression algorithm rather than some bleeding-edge implementation of a new one.)

The opposite situation should be avoided though - data provided by a VM being used as an input for decompression in dom0. That would be dom0 parsing a potentially malicious data structure, instead of treating it as just a bag of bytes that’s analyzed only for its statistical properties.

[disclose-mushy]

@rustybird

That would be with ZFS’s native encryption then, instead of LUKS?

Yes. I use raw send for authenticated incremental backups.

@rustybird

Have you experienced performance getting worse? Like for VM startup/shutdown/cloning/removal, or in-VM I/O?

When I recreate the pool again the UI feels snappier (browser tabs close and open faster), and VMs boot quicker.

I have set revisions_to_keep=-1 globally for all VMs in the varlibqubes pool, it made my dedup go from 20x to 1.20x once I deleted snapshots with it enabled. VMs boot/shutdown/clone much quicker with it disabled (I find it redundant because I use ZFS snapshots already).

[likeafox]

@rustybird

If compressing VM data in dom0 was judged to be a security risk, it would probably not be available at all in the backup system, let alone enabled by default…

I completely agree, so it’s fair to assume that compression was not judged to be a security risk for qvm-backup. Does that cause me to doubt that compressing qube volumes from within dom0 creates a fairly direct and realistic attack vector into dom0? Not really. I accept that I don’t know how to reconcile those things, and that it is difficult to try, without knowledge of the development process.

But that seems much less important when we’re able to consider the one truly necessary fact: Do compression functions have a reasonable potential for code execution vulnerabilities? The answer is yes. It happens, as in CVE-2018-25032 . And it is not unexpected, because they are often written in C, wherein arbitrary inputs influence code paths, pointer arithmetic, and heap management-- that is, just the kind of code expected to present a risk.

fwiw, we’ll agree that vulnerabilities related to decompression have posed a greater risk, and will continue to do so.

[rustybird]

Got to say, CVE-2018-25032 is a solid argument against compression. Not exactly a fancy new algorithm with some bleeding-edge implementation, is it. 13 years until that bug was found, and then another 4 years after the bugfix commit until that made it into packages. Ooof

@likeafox

it’s fair to assume that compression was not judged to be a security risk for qvm-backup.

@likeafox

I accept that I don’t know how to reconcile those things, and that it is difficult to try, without knowledge of the development process.

I was probably reading too much into the status quo there. Maybe it’s more accurate to say that compression was not judged to be a security risk at the time it became part of the backup system… :\

[qubematic]

@rustybird

Try running sudo filefrag on your VMs’ private.img files (and also root.img for TemplateVMs and StandaloneVMs) if you’re curious about their fragmentation. Assuming that this utility works on ZFS. (On Btrfs and XFS too IIRC, fragmentation starts being noticeable in the millions-of-extents range during VM startup/shutdown/etc.)

If I’m not mistaken, filefrag uses FIBMAP or FIEMAP ioctls, neither of which ZFS supports (there is a non-upstreamed FIEMAP patch out there). One could use zdb to get that information, though it requires knowing a bit about zfs internals..

[qubematic]

@disclose-mushy

ZFS users, do you tune ZFS for your system?

• Do you use dedup for your VMs?
• What recordsize do you use?
• What compression algo do you use?
• Which Qubes pool driver do you use, ZFS or file-reflink?
• How do you organize your datasets?
• How do you mitigate/handle zpool fragmentation?
• Do you use ZFS as your rootFS?

I’m mainly asking because I struggle with zpool fragmentation, but I’m also wondering how other users setup their systems.

These are fine and great questions. Would you be willing to share how you answered these for your setup? I’ve not committed to a setup yet, but here are some tentative answers:

• Do you use dedup for your VMs?
  • No, don’t plan on it, and think its generally a waste of resources. Offline dedup, OTOH, I think is more useful. Also, in this category is nop-write which requires compression and a crypto-secure hash. So I’m using blake3 (faster than sha-256) under the assumption that it won’t make a noticeable performance difference from the default, faster, non-crypto fletcher4. I’m not sure nop-write is that useful for my workload though. I’m also willing to take the crypto-hash hit because I believe that eventually I can use an offline dedup tool that uses those pre-existing hashes to find dupes, rather than having to rehash the data.
• What recordsize do you use?
  • Using default 16k
• What compression algo do you use?
  • auto configured, which should generally be lz4
• Which Qubes pool driver do you use, ZFS or file-reflink?
  • I would prefer reflink if the performance characteristics are not much worse because it carries the possibility of doing offline dedup (using FIDEDUPERANGE).
• How do you organize your datasets?
  • It seems to me that organization is basically set by the pool driver. Am I mistaken? Maybe there’s more leeway with the zfs pool driver, but with reflink, you can’t reflink across datasets. Or is this question about creating various pools for different datasets?
• How do you mitigate/handle zpool fragmentation?
  • Haven’t considered this as an issue (yet?)
• Do you use ZFS as your rootFS?
  • Not yet. It seems like a good idea. Although, I’m considering having dom0 root be in a different pool. This way if unrecoverable corruption happens in the qubes pool, disaster recovery can still be done from dom0. To that end, it would probably be good to have a backup of sys qubes in that pool as well.