The answer is in the post above yours:
" I just enabled xen_acpi_processor for R4.1"
No - its an HP EliteBook 840 G4 - just a standard PC, originally intended for Windows 10.
I ran the test for R4.1.0 using my i7-1065G7. The results weren’t as good as the original test on R4.0, but better than the R4.1 alpha tests.
I’ve updated the tables so all results that were run on the R4.1 Alpha ar clearly marked as such. Going forward, ‘R4.1’ refers to the stable release. I’ve also taken the liberty of removing minor version numbers for both kernels and versions since they just needlessly complicate things.
5.98
6.01
6.04
6.08
6.09
6.09
6.10
6.12
6.12
6.15
Mean: 6.08
Median: 6.09
Range: 5.98 - 6.15
Interesting thread, I wasn’t aware of it before.
But yes, 4.1 is slower according to my analysis as well [1], unfortunately for various reasons.
Actually I even wrote a tool just to measure it in more detail [2].
[1] 4.1 VM startup & qrexec performance issues · Issue #7075 · QubesOS/qubes-issues · GitHub
[2] GitHub - 3hhh/qubes-performance: Analyze Qubes OS VM startup performance.
1 Like
I’ve run several benchmark using fedora with several configuration, check here for interesting discussion
CPU : I7-10750H
Storage : WD SN 730 512GB
File System : LVM-XFS
Sector Size : 512b
Linux dom0 5.10.90-1.fc32.qubes.x86_64 #1 SMP Thu Jan 13 20:46:58 CET 2022 x86_64 x86_64 x86_64 GNU/Linux
# Boot speed
Startup finished in 4.897s (firmware) + 2.523s (loader) + 2.946s (kernel) + 8.787s (initrd) + 3.705s (userspace) = 22.861s
Startup finished in 4.868s (firmware) + 2.513s (loader) + 2.938s (kernel) + 8.817s (initrd) + 3.765s (userspace) = 22.902s
Startup finished in 4.874s (firmware) + 2.511s (loader) + 2.945s (kernel) + 8.255s (initrd) + 3.732s (userspace) = 22.318s
512b template fedora-34-full
# VM Boot
6.24
4.81
4.68
5.14
4.84
# Cryptsetup-reencrypt
Finished, time 15:24.011, 486745 MiB written, speed 526.8 MiB/s
CPU : I7-10750H
Storage : WD SN 730 512GB
File System : BTRFS+blake2b
Sector Size : 4kn
Linux dom0 5.10.90-1.fc32.qubes.x86_64 #1 SMP Thu Jan 13 20:46:58 CET 2022 x86_64 x86_64 x86_64 GNU/Linux
Startup finished in 4.898s (firmware) + 2.499s (loader) + 2.878s (kernel) + 7.922s (initrd) + 3.489s (userspace) = 21.688s
Startup finished in 4.878s (firmware) + 1.405s (loader) + 2.882s (kernel) + 7.936s (initrd) + 3.523s (userspace) = 20.626s
Startup finished in 4.889s (firmware) + 1.405s (loader) + 2.881s (kernel) + 7.817s (initrd) + 3.524s (userspace) = 20.518s
512b template fedora-34-full
#VM Boot
5.72
4.48
4.62
4.48
4.62
# directio
Finished, time 11:17.770, 486745 MiB written, speed 718.2 MiB/s
# no directio
Finished, time 12:32.743, 486745 MiB written, speed 646.6 MiB/s
CPU : I7-10750H
Storage : WD SN 730 512GB
File System : LVM+XFS
Sector Size : 4kn
Linux dom0 5.10.90-1.fc32.qubes.x86_64 #1 SMP Thu Jan 13 20:46:58 CET 2022 x86_64 x86_64 x86_64 GNU/Linux
Startup finished in 8.740s (firmware) + 2.472s (loader) + 2.947s (kernel) + 7.879s (initrd) + 3.588s (userspace) = 25.628s
Startup finished in 8.720s (firmware) + 2.469s (loader) + 2.947s (kernel) + 7.896s (initrd) + 3.679s (userspace) = 25.713s
Startup finished in 5.331s (firmware) + 2.479s (loader) + 2.947s (kernel) + 8.438s (initrd) + 3.619s (userspace) = 22.816s
512b template fedora-34-full
#VM Boot
5.75
4.60
4.59
4.61
4.59
4kn template fedora-35
#full-4096
3.77
3.89
4.03
3.86
3.90
#minimal-4096
3.68
3.67
3.62
3.79
3.68
#full-512b
3.62
3.70
3.82
3.58
3.62
# directio
Finished, time 13:27.041, 486745 MiB written, speed 603.1 MiB/s
# no directio
Finished, time 12:29.073, 486745 MiB written, speed 649.8 MiB/s
2 Likes
See also:
Not sure if its the CPU, but 3.x second VM startup times are impressive. I’m inclined to think its because you’re using Fedora since my i7-1065G7 is somewhat similar to yours. Are you planning on running the same test using Debian-11-minimal for comparison?
1 Like
The same thought occured to me but the test is only for Debian, so putting Fedora results there makes things confusing
First output is with all standard vms running.
Second output is after command shudown all
Lenovo Legion Y540-17IRH-PG0
Processor: i7-9750H 6Core 12Threads (2.6 GHz - 4.5 GHz) 12MB Cache Intel UHD Graphics 630
Chipset Memory: Mobile Intel HM370 Chipset 32GB max offering / 2666MHz DDR4, dual-channel capable, two DDR4 SO-DIMM sockets
Storage: 256 GB NVMe M.2 SSD 2280 PCIe 3.0 x 4, 32Gb/s
Some thoughts, do you have memory ballooning for dom0 or the domUs ?
What if you set mem=maxmem for dom0 and/or the debian template, to avoid it ?
Do you use VCPU pinning ? On my non-Qubes dom0s all the CPUs use the stock speed, never the Turbo one (but that may be my settings dunno).
Also, unman mentionned that the SSD was the most sensitive part about speed, so monitoring with iotop or even atop during the tests could help.
Maybe VCPU scheduling or memory/disk caching is at stake.
Monitoring with xentop and watch -n 1 xl vcpu-list during the tests may give pointers.
Are you interested in any tests or only bare metal ?
I run Qubes virtualized, and can change any setting at ease. Damn I just deleted the 4.0.4 domU …
Also, are you interested in a comparison with a “normal/non-Qubes” Xen install (but Debian dom0) ?
I can run the tests with a Debian netinst to compare.
left is debian, top right is 4kn fedora 35 minimal updated.
I’ll try build debian 4kn template and let see the difference.
I glanced through your thread and wasn’t able to parse out what I was looking for:
Is your test method based on the real output of dom0’s built-in time? I’m asking because I can’t wrap my head around how your CPU is starting even Debian with such speed. Did you overclock it, and by how much? Is the clock speed even the main difference here?
Basically the thread is opened by @Insurgo to tell us about :
Most of the ssd that i bought and know still use lbaf 0 which is 512b.
In the thread it’s explained that one need to change to lbaf 1 which use 4kb
for further reference see here Umstieg auf Advanced-Format-Festplatten mit 4K-Sektoren | Seagate Deutschland
After reading and some testing, in the same thread i post a workround how to run qubes with 4k support, changing the lbaf alone already made a significant performance improvement.
and in this benchmark, i share what the difference using a default (qubes os + 512b template provided by qubes) vs ( qubes os with 4k drive + 4k template builded by me)
I can’t say this is overclocking but it is tweaking.
it’s the same script we use in this thread, the difference is I set up only 5 times.
2 Likes
Thanks for taking the time to explain this to me. Since your setup is somewhat special I’ll go ahead and add your results and mark it SSD-4k.
All I know about SSD performance is SLC, MLC, QLC, etc. Maybe one day I can look forward to seeing ‘4K’ labels being abused on SSD packaging much like how everything had ‘HD’ labels at one point.
3 Likes
I’m astonished by the performance gained from using a 4k SSD with 4kn templates, and I’m willing to test it myself…
Could you please share details on how to build a 4kn debian template? I have looked at qubes-builder but seen no information on how to change the template’s sector size. Thanks in advance.
BTW, I don’t know why 512b fedora-35 is also booting so fast, almost the same as 4kn fedora-35, whereas 512b fedora-34 boots slower.
2 Likes
# Get the qubes builder
git clone https://github.com/QubesOS/qubes-builder
cd qubes-builder
# Use the base config already in the repo
cp example-configs/templates.conf builder.conf
# Append custom settings to builder.conf
# Better to update values in-place, but this is the 60 second version
cat >>builder.conf <<EOF
# Build Fedora 33; fc33+minimal or other flavors could be given
DISTS_VM = fc35+minimal
# Use the Qubes online repo for Qubes packages since we are not building them
USE_QUBES_REPO_VERSION = \$(RELEASE)
# Use the 'current' repo; not 'current-testing'
USE_QUBES_REPO_TESTING = 0
# Don't build dom0 packages
TEMPLATE_ONLY = 1
# Since this is TEMPLATE_ONLY, list the only components needed
TEMPLATE = builder-rpm linux-template-builder
EOF
# install build dependencies
make install-deps
# Download and verify sources (only Qubes git repos)
make get-sources-git
# Apply this patch
https://github.com/51lieal/qubes-linux-template-builder/commit/fe45b4f0c462590c95030f353bf0b32a26722466.patch
# Build the template
make template
You may want to edit prepare_image.sh and change xfs to ext4.
3 Likes
I got some interesting results.
On a SSD that only has 512bit LBA sector size supported (which means it cannot be changed to 4kn mode), I got better start time than my previous installation.
The new installation is on 4k-sector-sized xfs volume. I created it following @51lieal kindly provided guide. I cannot start any VM using the created “vm” lvm-thin pool, so I set default pool back to varlibqubes. My templates are 512b, I haven’t installed my 4kn templates.
My previous installation was on btrfs, using all default settings. 512b sector size, 512b templates, etc.
The specs of both SSD are mostly the same, except that my previous SSD has slightly better overall performance, and support two lbaf — one is 512 (default), the other is 4096 (unused).
2 Likes
I feel that while the presented numbers help describe the relative speed of different hardware, that the numbers are going to be deceptive for prospective Qubes users worrying about whether their systems are powerful enough for Qubes, since what they will actually care about is things like the human wait time for them to open a browser (I.E. whonix). which normally takes 30 seconds to several minutes.
Could you add some mention of how these numbers will be underestimates of their actual wait times?