Survey: CPU and VM boot time

General Discussion
15

System

Dom0 Kernel: 5.10.8-1
CPU: AMD Ryzen Embedded V1605B (4 Cores/ 8 Threads @2.0Ghz with Turbo @3.6Ghz)
Storage: Samsung 970 Evo Plus (M.2 NVMe PCIe 3.0)
RAM: 32 GB DDR4 2400 Mhz

Results

6.15
6.32
6.50
6.17
6.29
7.86.
6.59
6.54
6.29
6.17
----------
Median: 6.305
Mean: 6.488
Variance: 0.258

Method

I used a little script to obtain the values.

#!/usr/bin/bash

qube="debian-10-minimal"

get_real_time() {
  realtime="$(/usr/bin/time -f "%e" qvm-start -q ${qube})"
  qvm-shutdown --wait -q "${qube}"
  echo $realtime 
}

benchmark() {
  qvm-shutdown --all --wait -q
  for ((i = 0; i < 10 ; i++)); do
    sleep 15
    echo "$(get_real_time)"
  done
}

benchmark

1 Like
16

Thanks for sharing the script!

It’s good that it’s short enough to be manually typed into dom0 so we won’t have to ask people to paste things into it. I’ll add this as an option in the instructions.

Edit: I also went ahead and added your results while I was at it

1 Like
17

My more senior “stable” machine:

System

Dom0 Kernel: 5.4.88-1
CPU: Intel Core i7-3520M (2 Cores/ 4 Threads @2.9Ghz with Boost @3.6Ghz
Storage: Samsung SATA-III SSD
RAM: 8GB DDR3 1600Mhz

Results

7.05
8.37
8.99
6.51
8.95
7.84
7.34
9.11
8.12
9.35
----------
Median: 8.245
Mean: 8.163
Variance: 0.934
18

It turns out my previous test using i5-10210U wasn’t based on a 5.x kernel but a 4.19.147-1 kernel.

I’ve redone the test using a 5.6.16-1 kernel and @wind.gmbh 's handy dandy script. The results are as follows:

# s
1 5.25
2 5.41
3 6.19
4 5.99
5 4.40
6 4.32
7 5.90
8 6.22
9 6.29
10 4.47

Median: 5.7
Mean: 5.4
Range: 2.0

 

Manual testing:

# s
1 5.681
2 6.054
3 4.324
4 4.531
5 4.681
6 4.506
7 4.523
8 5.707
9 6.912
10 6.551

Median: 5.2
Mean: 5.3
Range: 2.6

Boot times exhibited a wider variance than using the 4.19 kernel. I also did manual testing (though not in a strictly consistent way) to see if the script affected times significantly. This does not seem to be the case. Based on this I can say that kernel 4.19 loads quicker and with less variance than 5.6.

2 Likes
19

Qubes 4.1
CPU: AMD A10-5750M
kernel-5.4.98-1.fc32.qubes.x86_64
SSD

11,76
11,80
11,63
11,77
11,57
11,50
11,43
11,45
11,58
11,60

1 Like
20

I finally got rid of R4.1 on my i7-1065G7 since the R4.0.4 installer works with its updated BIOS. I redid the tests and the results were much snappier:

debian-10-minimal (fresh download, not updated, kernel 5.10.8-1, using a modified version of wind.gmbh’s script):
4.08
4.18
4.16
3.89
7.21
4.14
3.92
7.53
4.05
4.01

Mean: 4.7
Median: 4.1
Range: 3.6

R4.0.4 is clearly faster than R4.1 when using the same kernel. There are some outliers, probably from some underlying processes, but otherwise the start times are nearly half that of R4.1. Anyone know what might be causing this?

3 Likes
21

My X230 i5 3230M with average VM boot time between 10 to 29 seconds running of a second hand laptop? Should I buy a new SSD to improve boot time?

22

I looked at the effect of RAM, CPU and Disk in x220 and x230 - the
biggest factor by far was the disk - definitely switch to a good SSD.

4 Likes
23

FIRST CHECK
real 0m16.531s
user 0m0.052s
sys 0m0.014s

SECOND CHECK
real 0m10.929s
user 0m0.043s
sys 0m0.022s

THIRD CHECK
real 0m12.041s
user 0m0.046s
sys 0m0.018s

model name: Intel® Core™ i5-8350U CPU @ 1.70GHz
Storage type: HDD full encrypted
VM Kernel version: 5.10.8-1
R 4.0.4

3 Likes
24

Definetly a weird observation that we should look into.

Without having done that my very first and very uneducated guess would be that different versions of Xen (4.8 → 4.14) could be the reason for that significant observation, because that is a significant difference that could influence the VM boot performance.
At least that does appear not unlikely to me, because Xen is mostly meant for servers, where it matters not that much whether a VM boot process takes 4 seconds or 8 seconds; so maybe not much effort is put into observing and keeping track of VM boot performance.

2 Likes
25

We should also look into whether this is something unique to my system or whether it’s a general issue. For example, would someone using an older CPU experience double their R4.0 boot times? E.g. would you experience a mean of 16 seconds on R4.1? That would have a significant quality-of-life impact and IMO should necessitate some developer action.

Maybe we should add another aspect to this survey–R4.1 vs R4.0

Edit: On second thoughts, maybe we should go ask people what version of Xen they’re using. I don’t know which would be better, so I’ll leave this up to a vote (via reply)

1 Like
26

For me, the avrage time were 7.9 s for R4.0.3 and 9.7 s for R4.1. But: R4.0.3 is running on an SSD with a SATA connection, and R4.1 on an SSD connected via USB. The SSDs themselves are the same model. So maybe the differences are not that significant and could be explained as being caused by the different connection types.

2 Likes
27

I should clarify for the record that both versions were run on the exact same hardware (default internal NVMe storage). Is it possible that your USB is fast enough (3.0 and above?) to eliminate the difference?

28

To add to unman: with SSDs so cheap nowadays and with performance gains so dramatic, upgrading storage should be the first place anyone on any platform should look at–it gives the most bang for the buck if you’re a HDD user.

You don’t even have to get a high capacity one–just stick all ‘chunky’ files like videos onto an HDD or thumb drive

29

Here for the stats are the result of my test inside vmware player on Win10.
I didn’t put it in the list as it runs inside a vm.

CPU: i7-4720HQ
HDD
Qubes 4.1

With Kernel 5.4:
12.5 sec
12.7 sec
12.6 sec
18 sec
12.4 sec

With Kernel 5.10
24 sec
30 sec

1 Like
30

My USB connection is 3.0, as far as the documentation tells.

31

Does 3.0 offer speeds fast enough to make an OS run as though it’s using SATA?

I’m asking because I have no idea. I know that latency shouldn’t be an issue for this test but there might be other factors

1 Like
32

I made several tests now and entered the results into the table above. The results for R4.0.4 are pretty close together. Neither the CPU type nor the type of disk connection (SATA or USB3.0) nor the kernel version make any big difference.

For R4.1, Qube startup is a lot (nearly 30 %) slower. The start times for R4.1 are close together and do not show the large variances that occured with R4.0.4:

9.21
9.38
9.34
9.24
9.18
9.22
9.16
9.31
9.00
9.01
Mean: 9.21
Median: 9.22
Variance: 0.02

5 Likes
33

Thanks for taking the time to test this. The next step would be to figure out if it’s the newer Xen that’s causing this. The easiest way to go about this is to install Xen 4.14 on R4.0.4 using --enablerepo=current-testing (I think). I’ll get around to it sometime this week.

That being said, I think the devs won’t worry too much about UX since the main purpose of the OS is security, with comfort an added bonus. Though if R4.1 pushes load times to unbearable levels for older computers, especially the X230, I think they might get on the case.

34

This is not really true. They even have a paid designer to improve the UX. Slow loading time is a UX-related bug like this one.