I have been testing the new 4K sector size in 4.3, and it’s a lot faster.
In 4.2 I’m getting ~4 GB/s in both dom0 and dumUs, and in 4.3 close to 13 GB/s in dom0 and 9 GB/s in domUs.
Is it expected that dom0 is that much faster than the domUs?
Dom0 kdiskmark
DomU kdiskmark
Wnat hardware did you use for testing?
The drive is a T705 x4 gen 5, CPU 9950X
Good question. Fortunately the current openQA tests include disk IO tests for dom0 and domIs.
Here are the latest results (for hw1 i.e. ProBook 445 G7):
| # terse_version_3 | jobname | read_kb | read_bandwidth_kb | read_iops | read_runtime_ms |
|---|---|---|---|---|---|
| dom0_root_seq1m_q8t1_read 3 | seq1m_q8t1_read | 1048576 | 390822 | 381 | 2683 |
| dom0_root_seq1m_q1t1_read 3 | seq1m_q1t1_read | 1048576 | 314038 | 306 | 3339 |
| dom0_root_rnd4k_q32t1_read 3 | rnd4k_q32t1_read | 406696 | 81306 | 20326 | 5002 |
| dom0_root_rnd4k_q1t1_read 3 | rnd4k_q1t1_read | 57576 | 11410 | 2852 | 5046 |
| dom0_varlibqubes_seq1m_q8t1_read 3 | seq1m_q8t1_read | 1048576 | 314982 | 307 | 3329 |
| dom0_varlibqubes_seq1m_q1t1_read 3 | seq1m_q1t1_read | 1048576 | 433474 | 423 | 2419 |
| dom0_varlibqubes_rnd4k_q32t1_read 3 | rnd4k_q32t1_read | 502224 | 100424 | 25106 | 5001 |
| dom0_varlibqubes_rnd4k_q1t1_read 3 | rnd4k_q1t1_read | 32436 | 6484 | 1621 | 5002 |
| fedora-42-xfce_root_seq1m_q8t1_read 3 | seq1m_q8t1_read | 1048576 | 413313 | 403 | 2537 |
| fedora-42-xfce_root_seq1m_q1t1_read 3 | seq1m_q1t1_read | 1048576 | 284784 | 278 | 3682 |
| fedora-42-xfce_root_rnd4k_q32t1_read 3 | rnd4k_q32t1_read | 448560 | 89676 | 22419 | 5002 |
| fedora-42-xfce_root_rnd4k_q1t1_read 3 | rnd4k_q1t1_read | 40188 | 8035 | 2008 | 5001 |
| fedora-42-xfce_private_seq1m_q8t1_read 3 | seq1m_q8t1_read | 1048576 | 363331 | 354 | 2886 |
| fedora-42-xfce_private_seq1m_q1t1_read 3 | seq1m_q1t1_read | 1048576 | 342001 | 333 | 3066 |
| fedora-42-xfce_private_rnd4k_q32t1_read 3 | rnd4k_q32t1_read | 452480 | 90459 | 22614 | 5002 |
| fedora-42-xfce_private_rnd4k_q1t1_read 3 | rnd4k_q1t1_read | 42556 | 8509 | 2127 | 5001 |
| fedora-42-xfce_volatile_seq1m_q8t1_read 3 | seq1m_q8t1_read | 1048576 | 343457 | 335 | 3053 |
| fedora-42-xfce_volatile_seq1m_q1t1_read 3 | seq1m_q1t1_read | 1048576 | 300882 | 293 | 3485 |
| fedora-42-xfce_volatile_rnd4k_q32t1_read 3 | rnd4k_q32t1_read | 391728 | 78204 | 19551 | 5009 |
| fedora-42-xfce_volatile_rnd4k_q1t1_read 3 | rnd4k_q1t1_read | 40872 | 8172 | 2043 | 5001 |
Ok. First of all, it appears to be a SATA drive. The maximum reported read bandwidth is ~433MB/s for varlibqubes pool sequential Q1T1 read. And for sure, it has a lot of variations:
| # terse_version_3 | jobname | read_bandwidth_kb |
|---|---|---|
| dom0_root_seq1m_q8t1_read 3 | seq1m_q8t1_read | 390822 |
| dom0_root_seq1m_q1t1_read 3 | seq1m_q1t1_read | 314038 |
| dom0_varlibqubes_seq1m_q8t1_read 3 | seq1m_q8t1_read | 314982 |
| dom0_varlibqubes_seq1m_q1t1_read 3 | seq1m_q1t1_read | 433474 |
| fedora-42-xfce_root_seq1m_q8t1_read 3 | seq1m_q8t1_read | 413313 |
| fedora-42-xfce_root_seq1m_q1t1_read 3 | seq1m_q1t1_read | 284784 |
| fedora-42-xfce_private_seq1m_q8t1_read 3 | seq1m_q8t1_read | 363331 |
| fedora-42-xfce_private_seq1m_q1t1_read 3 | seq1m_q1t1_read | 342001 |
| fedora-42-xfce_volatile_seq1m_q8t1_read 3 | seq1m_q8t1_read | 343457 |
| fedora-42-xfce_volatile_seq1m_q1t1_read 3 | seq1m_q1t1_read | 300882 |
I noticed that reducing the max cores in dom0 makes the disk IO drop as well, could some similar bottleneck that is happening in the domU
If you were on r4.3, you could have tried the same test with revisions_to_keep set to -1 to disable volume snapshot. CoW might be a contributor to performance degrade.
I have an ignorant question here…
Do the Qubes OpenQA tests happen sometimes/always on a bare xen hypervisor?
I tried reading about it, and it sounded a bit like it was in a nested configuration inside KVM, but I wasn’t sure at all.
It is another question to ask whether that would affect these tests!
Another interesting variable is the possibility that the writer/reader can use hugepages for large r/w.
I read somewhere that memory ballooning can prevent it due to fragmentation - even if it is not interesting to you to used fixed memory, it could falsify the benchmarks, if it was sometimes active and sometimes not.
I think it would also require large I/O blocksize (>2M?), and to be enabled in kernels(default?), but it seems xen supports it transparently, by this page at xenproject.org.
[Edit: I was thinking abou this after the recent “7z benchmark” thread… maybe there is some interest to compare multiple benchmarking methods]
[Edit2: note to self - does memory ballooning align allocation to hugepage boundaries, and could that make a performance difference? (… probably already taken care of!))
@alimirjamali
I’m using 4.3, settings revisions to -1 doesn’t seem to change anything.
Running kdiskmark uses a lot of CPU in the VM, I think the bottleneck is the fio process. Adding more cores does not seem to improve performance, so it could be limited by single thread performance.
@phceac
I have a lot of free memory, I don’t think Xen relocating any memory.
The tests happen on both virtualized KVM and on real hardware. More information here:
I have been testing the new 4K sector size in 4.3
Do you have a link to more information about this change/how to enable it once r4.3 drops?
https://github.com/QubesOS/qubes-issues/issues/4974
I don’t think you need to do anything to enable it.
