Sounds to me like someone set up their zpool with the default ashift value of 9 instead of 12.
Required reading:
ZFS on root is absolutely the future. I’m running it on Proxmox today and it works great. I only wish Qubes supported it; at a bare minimum, it would make snapshots and backups so much better.
Qubes OS supports it, it just doesn’t ship it.
I have guides on my blog to set it up.
Zvols are exactly what the Qubes ZFS driver uses. It avoids having to use any other layer on top. It’s really straightforward.
Thanks for the info and link. Really interesting.
The article I shared says nothing about ashift, but now I notice that the second link in it mentions that.
I also found some other discussion about it:
It would be great if we had a better clarification about these specifics in relation to Qubes.
@Rudd-O do you think you can probably explain these details in your guide? I guess most Qubes users use desktop systems with a single SSD, rather than enterprise setup, so write amplification is a common issue. What do you say?
-o ashift =12 is the way
And even on drives that claim to have 4k sectors rather than 512 bytes, I recall reading that this is still gonna be needed, and avoid such drives in the first place. It’s been a while, but IIRC people found that the 4k sector size was … a firmware thing and that in ZFS’s eyes they were still 512 byte drives, just badly behaved ones.
Definitely an area to research carefully. As in, I wouldn’t use a 4k drive unless I could afford on I might kill, and had a role for it where I could pummel it for several months, making sure it does what is expected.
Definitely an area to research carefully. As in, I wouldn’t use a 4k drive unless I could afford on I might kill, and had a role for it where I could pummel it for several months, making sure it does what is expected.
That’s why I am asking. These experiments are not cheap, so it would be quite unfortunate to damage a drive just to proof that ZFS is great. I am not saying it is not but rather that it may need special attention and clear documentation to be used correctly. It may turn out that for single nVME desktop systems the benefit from it may be marginal. Or it may turn out the opposite. I don’t know.
In any case, I am extra careful not to kill a 2-bit MLC drive which is more and more difficult to find these days.
In the case of Samsung it can be hard to figure out if the drive is MLC in the first place. It used to be the Pro models were, but IIRC they aren’t doing that consistently any more.
I am talking exactly about Samsung 970 Pro which is 2-bit MLC. On newer (and non-pro) models they use TLC and QLC.
My laptop is built on a Samsung 860 Pro 1TB…which I think is also MLC. Here’s the thing. Their own site won’t tell you!
I think the issue is Samsung decided Multi could mean whatever they wanted it to, when the term was originally invented to describe 2 bit.
I just looked through long third-party article explaining (over and over again) the differences between MLC and TLC, and it’s even supposed to be about Samsung SSDs, but it doesn’t say which Samsungs are MLC versus which ones are TLC.
(However the TBW rating for the 860 Pro is consistent with MLC (4,800 TBW, which is to say the thing can be written over 4800 times per block not 400 times per block).
Given this drive is about 4x the size I really need, its effective lifespan can be multiplied by 4 again.
Certainly in terms of lifespan an MLC is the economy bargain…ten or more times the life but less than ten times as expensive.
Some of the current models are… QLC (egad!).
ZFS will likely not damage a drive. Use 4K sectors because the drive internally is going to be writing larger swaths per sector anyway.
Good news everyone!
The guide to install ZFS on Qubes OS has been updated. It’s now simpler, thanks to deploy-zfs having gained native compatibility with Qubes OS 4.2.
Go check it out again:
My laptop is built on a Samsung 860 Pro 1TB…which I think is also MLC. Here’s the thing. Their own site won’t tell you!
A PDF on Samsung’s site says it is 2-bit MLC V-NAND.
ZFS will likely not damage a drive. Use 4K sectors because the drive internally is going to be writing larger swaths per sector anyway.
It would be good to have some factual comparison between the writes in case of ZFS vs e.g. ext4 in the same usage scenario. Do you know of any such comparison published anywhere?
Almost all of my data is stored in veracrypt containers. Obviously it would be beneficial to set those up as zfs (If I can figure out how to just have one plain drive be a pool, etc., so it just looks like a disk. I have forgotten the relevant terminology unfortunately.)
Veracrypt doesn’t natively format disks zfs so I’d have to basically decrypt the container and reformat it myself…and I’d need to have zfs installed on every VM that might have one of these containers mounted on it.
This might be an interesting thing to tinker with.
This would be totally different from converting Qubes OS over to zfs as you outline on your site. (That’s actually less of a priority for me because almost all of my qubes have no real data on them; if one of them got trashed I can just rebuild it. The ones that do have data, I back up regularly.)
Another crazy thought: zfs on a thumb drive. (It’s entertaining enough that I typically reformat them in ext4, all one partition.)
I met with success getting veracrypt containers to use zfs.
Veracrypt will not make a zfs container. You have to tell veracrypt to make it something else then adjust it later. That means decrypt the container, but unmount it. Then go to the device and format it as zfs (single device pool). That of course will automatically import what you just made, so you will want to export before having veracrypt close the container. (And for some weird reason I have to issue the export command twice.)
The main thing to do, when decrypting the container in the future, is tell veracrypt not to mount the decrypted file system. You do this with --filesystem=none on the command line. You then import the pool. (In the veracrypt GUI, when “mounting” the container, you can, before entering the password, expand Options in the lower right, and just above the bottom check the box that reads “do not mount filesystem.”)
Actually my situation is a lot more complicated because I am using split-veracrypt. In my case I had to tell the decryptor qube --filestystem=none but I should have been doing that all along (it saves me unmounting the filesystem to turn it into a loop; it’s a bit more secure because the decryptor doesn’t have even momentary access to what it is decrypting).
There were various other complications in split-veracrypt, but the point is, you CAN veracrypt a zfs file system (single device).
Q: Why not directly use ZFS encryption in your case? The pool metadata would be plainly visible, but anything inside the file systems (including file or extent metadata) would not be.
Good timing!
I don’t understand ZFS encryption; basically when I bought my NAS the documentation did everything but say DON’T USE THIS. More trouble than its worth, password had to be stored in the clear on the NAS anyway, etc.
Of course that could be the NAS not ZFS. In any case, I’m fairly well acquainted with veracrypt. And with split veracrypt I can decrypt in one qube and mount/use in another, with the mount qube never seeing the password.
I just spent several hours making a rather alarming discovery: If I install network support (i.e., the ability to connect to a network qube) 8and* ZFS in the same qube…the qube will die when I try to start it. I haven’t the foggiest idea why.
So, until I figure that one out, no ZFS on any qube that talks to the internet over wifi. No issue here, none of them have significant storage on them.
The ethernet qubes have the same issue, and they talk to my NAS–where all of these containers are. But even here I can work around it. Those qubes create containers on the NAS, and also hand still-encrypted containers off to the decryptor in my split veracrypt scheme. In other words, they don’t have to see ZFS either. (Admittedly it’s a bit inconvenient to have to hand a container off to some other qube in order to format it ZFS; it would have been cool if I could do that in the same place that created the container.)
I tried deploy_zfs in your script in dom0 and got an error:
qvm-run: error: no such domain: 'fedora-user-tpl-dvm'
I found you specify fedora-user-tpl-dvm as dispvm in cmd.py, while vm with that name does not exist in my system.
What is fedora-user-tpl-dvm?
That’s strange! I will fix that shortly. I hardcoded a VM there.
Can you please check the latest commit and see if it works for you there?