Survey: CPU and VM boot time

General Discussion
21

My X230 i5 3230M with average VM boot time between 10 to 29 seconds running of a second hand laptop? Should I buy a new SSD to improve boot time?

22

I looked at the effect of RAM, CPU and Disk in x220 and x230 - the
biggest factor by far was the disk - definitely switch to a good SSD.

4 Likes
23

FIRST CHECK
real 0m16.531s
user 0m0.052s
sys 0m0.014s

SECOND CHECK
real 0m10.929s
user 0m0.043s
sys 0m0.022s

THIRD CHECK
real 0m12.041s
user 0m0.046s
sys 0m0.018s

model name: IntelĀ® Coreā„¢ i5-8350U CPU @ 1.70GHz
Storage type: HDD full encrypted
VM Kernel version: 5.10.8-1
R 4.0.4

3 Likes
24

Definetly a weird observation that we should look into.

Without having done that my very first and very uneducated guess would be that different versions of Xen (4.8 ā†’ 4.14) could be the reason for that significant observation, because that is a significant difference that could influence the VM boot performance.
At least that does appear not unlikely to me, because Xen is mostly meant for servers, where it matters not that much whether a VM boot process takes 4 seconds or 8 seconds; so maybe not much effort is put into observing and keeping track of VM boot performance.

2 Likes
25

We should also look into whether this is something unique to my system or whether itā€™s a general issue. For example, would someone using an older CPU experience double their R4.0 boot times? E.g. would you experience a mean of 16 seconds on R4.1? That would have a significant quality-of-life impact and IMO should necessitate some developer action.

Maybe we should add another aspect to this surveyā€“R4.1 vs R4.0

Edit: On second thoughts, maybe we should go ask people what version of Xen theyā€™re using. I donā€™t know which would be better, so Iā€™ll leave this up to a vote (via reply)

1 Like
26

For me, the avrage time were 7.9 s for R4.0.3 and 9.7 s for R4.1. But: R4.0.3 is running on an SSD with a SATA connection, and R4.1 on an SSD connected via USB. The SSDs themselves are the same model. So maybe the differences are not that significant and could be explained as being caused by the different connection types.

2 Likes
27

I should clarify for the record that both versions were run on the exact same hardware (default internal NVMe storage). Is it possible that your USB is fast enough (3.0 and above?) to eliminate the difference?

28

To add to unman: with SSDs so cheap nowadays and with performance gains so dramatic, upgrading storage should be the first place anyone on any platform should look atā€“it gives the most bang for the buck if youā€™re a HDD user.

You donā€™t even have to get a high capacity oneā€“just stick all ā€˜chunkyā€™ files like videos onto an HDD or thumb drive

29

Here for the stats are the result of my test inside vmware player on Win10.
I didnā€™t put it in the list as it runs inside a vm.

CPU: i7-4720HQ
HDD
Qubes 4.1

With Kernel 5.4:
12.5 sec
12.7 sec
12.6 sec
18 sec
12.4 sec

With Kernel 5.10
24 sec
30 sec

1 Like
30

My USB connection is 3.0, as far as the documentation tells.

31

Does 3.0 offer speeds fast enough to make an OS run as though itā€™s using SATA?

Iā€™m asking because I have no idea. I know that latency shouldnā€™t be an issue for this test but there might be other factors

1 Like
32

I made several tests now and entered the results into the table above. The results for R4.0.4 are pretty close together. Neither the CPU type nor the type of disk connection (SATA or USB3.0) nor the kernel version make any big difference.

For R4.1, Qube startup is a lot (nearly 30 %) slower. The start times for R4.1 are close together and do not show the large variances that occured with R4.0.4:

9.21
9.38
9.34
9.24
9.18
9.22
9.16
9.31
9.00
9.01
Mean: 9.21
Median: 9.22
Variance: 0.02

5 Likes
33

Thanks for taking the time to test this. The next step would be to figure out if itā€™s the newer Xen thatā€™s causing this. The easiest way to go about this is to install Xen 4.14 on R4.0.4 using --enablerepo=current-testing (I think). Iā€™ll get around to it sometime this week.

That being said, I think the devs wonā€™t worry too much about UX since the main purpose of the OS is security, with comfort an added bonus. Though if R4.1 pushes load times to unbearable levels for older computers, especially the X230, I think they might get on the case.

34

This is not really true. They even have a paid designer to improve the UX. Slow loading time is a UX-related bug like this one.

35

Iā€™m pretty bummed. I have an HP Z8 G4 which I originally bought specifically to run Qubes, but ultimately settled on an Intel NUC as my Qubes-horse over the last few years. I just dusted the HP off and installed 4.0.4 on it. It is a dual CPU system with moderately specced XEON CPUs

Model name: Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz

but with 64GB ECC memory and 4 500GB NVMe PCIe attached SSDs.
Each CPU has 10 cores, and I have HT off, but with HT on I could have 40 cores total, rather than 20 total.

I am just using 2 of the 4 500G SSDs for Qubes, with btrfs handling the two luks encrypted devices. Under btrfs, data are raid0 and metadata are raid1, so reads should be PDQ. Running the script I get 10.9 average time, with a wild variance. Min was 6.8, max was 16. Yes, all VMs shut down.

I had been feeling that this was a bit less snappy than I was hoping. It takes forever to boot compared to the NUC. Perhaps I need to try without btrfs.

Maybe we need a perf test where we spin up N disposable VMs from the debian minimal, and use qvm-run to run ls -Rl / or something. Or give the template the same VCPU as the physical core count and qvm-run /bin/some-embarrassingly-parallel-programme to validate my choices.

Running fedora33 workstation with root on btrfs on the other two 500GB NVMe, also on luks encrypted devices, is indeed nice and snappy.

3 Likes
36

As a non-technical person I donā€™t have a clue about btrfs, raid, and PDQ, but welcome the idea of scripts running variants of the VM boot test.

Since results from these tests are collated using wiki-posts, it might be easier to have separate threads for each major variant. If there are enough variants, maybe group them under a separate portal thread or a new topic label.

Iā€™m also curious about the cause of massive variances in some systems (usually not R4.1). Even though dom0 should be the only thing running (and thereā€™s usually little going on in its background AFAIK), I saw spikes of nearly double my previous boot times in R4.0.4. Does anyone have any idea?

Ā 

Scrolling up as Iā€™m typing this, I just noticed @GWeck just added a trove of data to the wiki. Thank you!

37

Things are getting slightly better.
I diddled with the BIOS settings:

  • enabled Turbo mode
  • turned on NUMA, and selected some more performant NUMA setting
  • I think there was one other setting I enabled.

Now I am getting:

t0=8.87
t1=8.83
t2=8.61
t3=9.92
t4=9.61
mu=9.168 var=0.319 sigma=0.565

t0=6.18
t1=8.92
t2=9.47
t3=9.52
t4=8.47
mu=8.512 var=1.885 sigma=1.373

I made some adjustments to the script that was posted here, and placed it in
github

It is a work in progress. I intend on adding some other tests. Ignore the xentrace stuff for now. I actually had my best times to date when I was collecting xentrace data at the same time, but figuring out what to do with itā€¦

By the way, the script can be slurped into dom0 using this handy script:

dom0# cat bin/passio
Usage=ā€œpassio vm srcā€
VM=${1?$Usage}
SRC=${2?$Usage}

qvm-run --pass-io ${VM} ā€œcat ${SRC}ā€

As in bin/passio fromVM /home/user/Downloads/bench >./bench

1 Like
38

Here are some results from an Intel NUC7i5BNH.

i5-7260U @ 2.20GHz
kernel: 5.4.88-1
16GB mem

t0=5.19
t1=6.90
t2=7.42
t3=8.70
t4=6.46

mu=6.934 var=1.656 sigma=1.287

1 Like
39

Just assume that this might be of interest to the the people here: Regarding the performance issues in R4.1 @Jarrah gave the useful hint that this is about the CPU running on slower frequencies in Xen.

1 Like
40

@GWeck Case closed?