Survey: CPU and VM boot time

I have run the tests again with xen_acpi_processor enabled:

System

Dom0 Kernel: 5.10.21-1
CPU: AMD Ryzen Embedded V1605B (4 Cores/ 8 Threads @2.0Ghz with Turbo @3.6Ghz)
Storage: Samsung 970 Evo Plus (M.2 NVMe PCIe 3.0)
RAM: 32 GB DDR4 2400 Mhz (ECC)
Release: R4.1

Results

6.80
6.07
6.57
6.32
6.80
6.18
6.74
6.16
6.14
6.78
----------
Median: 6.445 (was: 6.305)
Mean: 6.456 (was: 6.488)
Variance: 0.096 (was: 0.258)

(Looking at the frequencies, I realized that it probably worked before.)

2 Likes

I presume itā€™s for R4.1?

If so, thereā€™s probably something else slowing it down. I didnā€™t modify my xen_acpi_processor and I doubt @GWeck did either.

Since I no longer have R4.1, I sadly canā€™t test this theory

Thatā€™s of course on R4.1. I have added that fact to my post, because I wrongly presumed that this was implied.

Maybe AMD works better/ different than Intel?

I knewā€“just wanted to be explicit for the sake of clarity. Thanks for fixing that

2 Likes

Given Intelā€™s general poor performance these years, it wouldnā€™t surprise me.

I just enabled xen_acpi_processor for R4.1 - thanks to @gust .Major known 4.1 gaps? - and got the following values:

7,34
7,54
7,61
7,45
7,34
7,25
7,55
7,44
7,57
7,20
Mean: 7.43
Median: 7.45
Variance: 0.02

Thatā€™s pretty close to the values for R4.0, but without the large variances.

4 Likes

@GWeck

Do you have it running on a raspberry pi?? I know xen has already been ported and deployed for the raspberry pi, but i didnā€™t know anyone had ported qubes to it.

The answer is in the post above yours:
" I just enabled xen_acpi_processor for R4.1"

No - its an HP EliteBook 840 G4 - just a standard PC, originally intended for Windows 10.

5 posts were split to a new topic: Canā€™t Search Two-character Words on Forum

I ran the test for R4.1.0 using my i7-1065G7. The results werenā€™t as good as the original test on R4.0, but better than the R4.1 alpha tests.

Iā€™ve updated the tables so all results that were run on the R4.1 Alpha ar clearly marked as such. Going forward, ā€˜R4.1ā€™ refers to the stable release. Iā€™ve also taken the liberty of removing minor version numbers for both kernels and versions since they just needlessly complicate things.

5.98
6.01
6.04
6.08
6.09
6.09
6.10
6.12
6.12
6.15

Mean: 6.08
Median: 6.09
Range: 5.98 - 6.15

Interesting thread, I wasnā€™t aware of it before.

But yes, 4.1 is slower according to my analysis as well [1], unfortunately for various reasons.

Actually I even wrote a tool just to measure it in more detail [2].

[1] 4.1 VM startup & qrexec performance issues Ā· Issue #7075 Ā· QubesOS/qubes-issues Ā· GitHub
[2] GitHub - 3hhh/qubes-performance: Analyze Qubes OS VM startup performance.

1 Like

Iā€™ve run several benchmark using fedora with several configuration, check here for interesting discussion

CPU : I7-10750H
Storage : WD SN 730 512GB
File System : LVM-XFS
Sector Size : 512b

Linux dom0 5.10.90-1.fc32.qubes.x86_64 #1 SMP Thu Jan 13 20:46:58 CET 2022 x86_64 x86_64 x86_64 GNU/Linux

# Boot speed
Startup finished in 4.897s (firmware) + 2.523s (loader) + 2.946s (kernel) + 8.787s (initrd) + 3.705s (userspace) = 22.861s
Startup finished in 4.868s (firmware) + 2.513s (loader) + 2.938s (kernel) + 8.817s (initrd) + 3.765s (userspace) = 22.902s
Startup finished in 4.874s (firmware) + 2.511s (loader) + 2.945s (kernel) + 8.255s (initrd) + 3.732s (userspace) = 22.318s

512b template fedora-34-full
# VM Boot 
6.24
4.81
4.68
5.14
4.84

# Cryptsetup-reencrypt
Finished, time 15:24.011, 486745 MiB written, speed 526.8 MiB/s
CPU : I7-10750H
Storage : WD SN 730 512GB
File System : BTRFS+blake2b
Sector Size : 4kn

Linux dom0 5.10.90-1.fc32.qubes.x86_64 #1 SMP Thu Jan 13 20:46:58 CET 2022 x86_64 x86_64 x86_64 GNU/Linux

Startup finished in 4.898s (firmware) + 2.499s (loader) + 2.878s (kernel) + 7.922s (initrd) + 3.489s (userspace) = 21.688s
Startup finished in 4.878s (firmware) + 1.405s (loader) + 2.882s (kernel) + 7.936s (initrd) + 3.523s (userspace) = 20.626s
Startup finished in 4.889s (firmware) + 1.405s (loader) + 2.881s (kernel) + 7.817s (initrd) + 3.524s (userspace) = 20.518s

512b template fedora-34-full
#VM Boot
5.72
4.48
4.62
4.48
4.62

# directio
Finished, time 11:17.770, 486745 MiB written, speed 718.2 MiB/s

# no directio
Finished, time 12:32.743, 486745 MiB written, speed 646.6 MiB/s
CPU : I7-10750H
Storage : WD SN 730 512GB
File System : LVM+XFS
Sector Size : 4kn

Linux dom0 5.10.90-1.fc32.qubes.x86_64 #1 SMP Thu Jan 13 20:46:58 CET 2022 x86_64 x86_64 x86_64 GNU/Linux

Startup finished in 8.740s (firmware) + 2.472s (loader) + 2.947s (kernel) + 7.879s (initrd) + 3.588s (userspace) = 25.628s
Startup finished in 8.720s (firmware) + 2.469s (loader) + 2.947s (kernel) + 7.896s (initrd) + 3.679s (userspace) = 25.713s
Startup finished in 5.331s (firmware) + 2.479s (loader) + 2.947s (kernel) + 8.438s (initrd) + 3.619s (userspace) = 22.816s 

512b template fedora-34-full
#VM Boot
5.75
4.60
4.59
4.61
4.59

4kn template fedora-35
#full-4096
3.77
3.89
4.03
3.86
3.90

#minimal-4096
3.68
3.67
3.62
3.79
3.68


#full-512b
3.62
3.70
3.82
3.58
3.62

# directio
Finished, time 13:27.041, 486745 MiB written, speed 603.1 MiB/s

# no directio
Finished, time 12:29.073, 486745 MiB written, speed 649.8 MiB/s
2 Likes

See also:

Not sure if its the CPU, but 3.x second VM startup times are impressive. Iā€™m inclined to think its because youā€™re using Fedora since my i7-1065G7 is somewhat similar to yours. Are you planning on running the same test using Debian-11-minimal for comparison?

1 Like

Impressive indeed. @51lieal do you think you can add these results to the table on the first post?

The same thought occured to me but the test is only for Debian, so putting Fedora results there makes things confusing

First output is with all standard vms running.
Second output is after command shudown all

Lenovo Legion Y540-17IRH-PG0
Processor: i7-9750H 6Core 12Threads (2.6 GHz - 4.5 GHz) 12MB Cache Intel UHD Graphics 630
Chipset Memory: Mobile Intel HM370 Chipset 32GB max offering / 2666MHz DDR4, dual-channel capable, two DDR4 SO-DIMM sockets
Storage: 256 GB NVMe M.2 SSD 2280 PCIe 3.0 x 4, 32Gb/s

Screenshot_2022-04-07_19-37-01

Some thoughts, do you have memory ballooning for dom0 or the domUs ?
What if you set mem=maxmem for dom0 and/or the debian template, to avoid it ?
Do you use VCPU pinning ? On my non-Qubes dom0s all the CPUs use the stock speed, never the Turbo one (but that may be my settings dunno).
Also, unman mentionned that the SSD was the most sensitive part about speed, so monitoring with iotop or even atop during the tests could help.
Maybe VCPU scheduling or memory/disk caching is at stake.
Monitoring with xentop and watch -n 1 xl vcpu-list during the tests may give pointers.

Are you interested in any tests or only bare metal ?
I run Qubes virtualized, and can change any setting at ease. Damn I just deleted the 4.0.4 domU ā€¦
Also, are you interested in a comparison with a ā€œnormal/non-Qubesā€ Xen install (but Debian dom0) ?
I can run the tests with a Debian netinst to compare.

left is debian, top right is 4kn fedora 35 minimal updated.

Iā€™ll try build debian 4kn template and let see the difference.