Pool level deduplication?

[Demi]

@Insurgo

@Insurgo

ZFS : native pool level deduplication exists and is applied on live systems

• Deduplication Table (DDT)
• seems really effective
• Cost for deduplication is between 1-2gb of ram per 1tb of storage

I agree with @Rudd-O:

@Rudd-O

Re deduplication: dedup requires very little memory vis a vis the amount of memory Qubes OS users have and storage size they have

If we talk about a 1tb of non-deduplicated storage space that consumes 1gb ram, i would use that. If backuping/restoring would only also take dedupped space, I would also use that.

That is a very valid trade-off (to each their own ) and would definitely justify supporting it, especially on high-end systems with 32GiB or more of RAM.

@Insurgo

On VDO and what I understood of it, it is deduplicating on block (filesystem) layer, not pool-wide:

@Insurgo

VDO on LVM : unfit for template

• This technology is aimed at removing duplicates inside of a same LVM. Multiple copies of an ISO will not consume multiple times its size.

VDO is block-layer deduplication. One puts it below filesystems or LVM thin provisioning, and above encryption and/or RAID. Therefore, it works across volumes. One would need to patch LVM to allow VDO to be used as a thin-pool data volume, however, as LVM does not allow this because of out-of-space recovery problems. Recovery would be very difficult and would at a minimum require adding more storage to the VDO volume, which can never be removed.

@Insurgo

Author recommends optane SSDs:

Optane and pure battery backed RAM cards only .
I should clarify: That’s nothing to do with SSDs having too-small DRAM or SLC cache. It’s inherent in the SSD NVRAM chips themselves. Because it’s nothing to do with the device cache type or size, a “better” SSD or one with “better” or no cache, won’t help much.

If this statement is accurate, then I do not believe the Qubes team can recommend deduplication. The Optane SSDs used by the author of that post have been discontinued, and the only pure-X-Point storage available is the enterprise drives. Those are extremely expensive, so I doubt they will be an option for the majority of people.