I don’t get the question. What does fine mean? And aren’t you benchmarking 512-byte sectors on XFS/file-reflink Oh, your varlibqubes, vs. 4K sectors on LVM Thin vm-pool (using IIUC a custom partitioned TemplateVM root volume) - which would be two very different storage drivers?vm-pool is XFS/file-reflink on top of LVM Thin? Okay that would be the same Qubes storage driver then, but it’s still a different (and unusual) storage stack.
@rustybird: This is really interesting!!! Please poke me on updates of this. Won’t land under Qubes before next release for sure, but this is really pertinent advancement in my opinion, even more if applying to default partition scheme (thin lvm, seperated root/vm pools).
I recently installed qubes exactly as what you have described, creating partitions, specifying qvm-pool, manually installing templates, etc. It all worked well, and I’m grateful for your instructions.
However, I could’t get any of VM to start. In their log, I saw that they complained about the filesystem of /xvdc, as you have described in that GitHub issue. I think that line of qvm-pool command was intentioned to avoid this ( by using lvm thin pool, as you said on GitHub), but unluckily it didn’t work for me.
Should I reinstall qubes, or should I build 4kn templates and find a way to transfer them into dom0 without any VM running? Thank you!
Btw, my self-built 4kn template also fails to start for the same reason, in qubes on a 512e ssd.
Well i need more detailed what step by step you have done, but let’s see next week, I’ll try to create a guide from changing lbaf to using 4k template.
Only this pull request:
@rustybird Sorry I was not more specific: I meant for the root and private volumes creation: was that tested working?
So if I understand well, I could apply your patch and have volatile volume fixed. But for creating root volumes and private volumes, I would need to build ISO, or patch stage 1 and stage 2 install so that when templates are decompressed, those are fixed to create a working system to be able to compare performance properly with/without the fixes.
I was looking for next steps to get main devs attention in seeing actual performance losses/ differences in this thread.
Otherwise, people are trying to get away of LVM thin provisioning model at install as of now. Some wants ZFS,XFS/BRTFS since speed differences are quite important.
One example of that is from @Sven at https://forum.qubes-os.org/t/ext4-vs-btrfs-performance-on-qubes-os-installs as an example of that, showing gains of ~300mb/s write speed by choosing BRTFS at install vs thin provisioning default:
Fixing LUKS+LVM thin provisioning would be great. Otherwise LVM is blamed for performance losses as of now where other implementations are simply not suffering from the same implementation flaws that LVM thin provisioning is suffering from, per Qubes implementation of volatile, private and root volumes creation.
@Demi maybe? I think @rustybird showed where love is needed here: SSD maximal performance : native sector size, partition alignment - #30 by rustybird
1 Like
Not sure that I understand your question, but standard (i.e. not in like a standalone HVM) private volumes are already sector-size agnostic in their content, so compatibility wise it doesn’t matter whether they are presented to the VM as 512B or 4KiB block devices.
Standard root volumes have sector-size specific content, and I don’t think it’s feasible to dynamically patch that volume content (specifically, the partition table) in dom0, because it contains untrusted and potentially malicious VM controlled data.
Backward compatibility is a real headache here. It seems like the existing root and private volumes should simply be presented to the VM as 512B devices by default for now. In the case of an LVM installation layout, that might even entail forcing 512B sectors for the whole LUKS device - unless there’s a good way to set an independent sector size for the LVM pool or ideally per LVM volume.
Cross-referencing important post by @tasket (one filesystem knowledgeable person with a lot of hands on experiment, behind wyng):
I’d like to share some info about my playing with the 4kn drive.
I went through 51lieal’s instructions (using stock r4.1.1 iso). I got two main volumes, one is “varlibqubes” using file-reflink driver, the other is “vm” using lvm-thin driver.
When I was installing templates, I found that stock 512b templates didn’t boot on “vm”, but booted on “varlibqubes”. My self-built 4kn template booted on both two volumes.
However, since I upgraded dom0 to testing-latest, all my VMs refuse to boot, and the error messages are “libxenlight failed to create domain xxx”, just like those when I try to boot 512b templates on “vm”.
I can confirm this isn’t a kernel issue, because downgrading kernel version doesn’t help. I suspect this is somthing xen related.
Edit：
Even stock “LVM” installation ( because I cannot proceed with “LVM thin” option) without those modifications to luks on a 4kn drive will lead to the same result, that VMs refuse to boot when testing-repos are enabled.
Was that tested with xmm-xen from trsting repos? (With without directio patch?)
I am confused since that patch was removed in latest xmm-xen, which is why I’m asking.
Sorry just saw that this issue was referred under VMs don't boot on 4Kn drive · Issue #7828 · QubesOS/qubes-issues · GitHub
Important discussions happening there.
Important advancements here VM partition layout expects 512B sector size, VMs fail to start from 4096B drive · Issue #4974 · QubesOS/qubes-issues · GitHub
Props to @rustybird ! Thanks!!!
What is your SSD? 102mb/s for cryptsetup-reencrypt seems a bit on the lower end of speed?
Eh, it’s an SATA Samsung 850 Pro on an old ThinkPad T420. 
I’ve not using xfs+lvm-thin anymore so i don’t have idea, currently using btrfs, and there’s no problem with it (i’m using testing repo).
Yes, Yes, Marmarek just reverted “Use direct-io for loop device” in xen 4.14.5-10, which should be in testing repo now.
$ cat /sys/block/nvme0n1/queue/physical_block_size
512
and
$ cat /sys/block/nvme1n1/queue/physical_block_size
512
QubesOS 4.1 on two nvme SSD’s. Disk model WDC WDS500G2B0C-00PXH0 (x2)
Notebook model: NH5x_NH7xHP serial: N/A UEFI: INSYDE v: 1.07.01
CPU: Single Core 11th Gen Intel Core i7-11800H speed: 2304 MHz
Extreme slow boot, something like 6 minutes:
@dom0 ~]$ systemd-analyze blame
49.894s qubes-vm@sys-firewall.service >
43.064s qubes-vm@sys-usb.service >
32.900s dracut-initqueue.service >
31.298s systemd-cryptsetup@luks\x2d05b49888\x2df626\x2d47c1\x2d9dde\x2d26bfd00f>
28.157s qubes-vm@sys-net.service >
27.710s systemd-cryptsetup@luks\x2daeb29ae5\x2d9b10\x2d4de2\x2d8665\x2d579831c5>
15.741s qubesd.service >
15.663s upower.service >
15.569s systemd-logind.service >
8.355s systemd-udev-settle.service >
7.327s lvm2-monitor.service >
4.793s plymouth-quit-wait.service >
3.263s lvm2-pvscan@253:0.service >
2.218s systemd-journal-flush.service >
1.680s dracut-cmdline.service >
840ms qubes-qmemman.service
@dom0 ~]$ systemd-analyze critical-chain
The time when unit became active or started is printed after the “@” character.
The time the unit took to start is printed after the “+” character.
graphical.target @1min 18.259s
└─multi-user.target @1min 18.258s
└─qubes-vm@sys-firewall.service @28.363s +49.894s
└─qubes-meminfo-writer-dom0.service @28.354s
└─qubes-core.service @27.996s +330ms
└─qubesd.service @12.227s +15.741s
└─lvm2-pvscan@253:0.service @8.957s +3.263s
└─system-lvm2\x2dpvscan.slice @8.891s
└─system.slice
└─-.slice
I am just a user, when this issue is solved by the tech-people I’ll just reïnstall Qubes en restore from back-up.
Or stop using Qubes on this machine because another issue is I really need to install rpmfusion repos in dom0, what is not available in EOL Fedora 32.
Oh! That was online reencryption, never did that!
I lost track on github issues with last comment being that open-qa fails as expected at https://github.com/QubesOS/qubes-issues/issues/4974#issuecomment-1295866835
What are the next steps @rustybird ?
In link with your comment VMs don't boot on 4Kn drive · Issue #7828 · QubesOS/qubes-issues · GitHub
Edit: sorry. I think as I go and seem to not be able to do a full post one shot and always edit multiple times. Sorry if you reply from email, hopefully this is sent to you in the 10 minutes edition permission time prior of sending the email you would reply to.
It’s pretty neat.
I’m not sure. For the time being, people with 4Kn drives should probably simply install Qubes OS with the Btrfs installation layout.
@rustybird unfortunately, not a desirable path for everything thin-lvm related support, including wyng support 
1 Like