I had the same reaction, but maybe @fiftyfourthparallel means from power on to browser open (including booting the entire system)?
The results are for starting a minimal template with no other qubes
running: just starting.
@ddevz seems to be talking about opening a browser - presumably from
selecting from the menu to the browser opening, but it isn’t clear what
other qubes might be involved in that process. (sys-net? sys-firewall?
Whonixes?)
Whatever they might be, I’d recommend identifying the hardware so we can
maintain a blacklist as well as your useful white list.
You probably mean me, not fiftyfourthparallel, and no, I mean:
Prep the system: (I.E. select a disposable whonix browser from the start button, wait for sys-whonix to start, after the browser appears, close the browser (this just make sure were not waiting for other VMs to start during the timed test))
Do the timed test: start the timer as you select a disposable whonix browser from the start button, stop the timer the moment the web browser appears.
I’ve seen it across multiple systems. Here’s what I’ve seen:
Normal system with a HDD, on either qubes 4.0 or 4.1:
browser wait time: several min
A system specifically bought just to run qubes faster then a normal system:
- Intel(R) Xeon(R) Gold 6240R CPU @ 2.40GHz
- VM kernel version: 5.10
- M2 module (I.E. solid state)
- Qubes 4.1
- Memory: 128 Gigs
browser wait times:
24.5 seconds
24.5 seconds
25.3 seconds
doing the “debian-11 minimal” procedure:
7.2 seconds
7.4 seconds
7.6 seconds
7.2 seconds
same thing but with a up to date debian-11 image:
6.5 seconds
6.4 seconds
6.7 seconds
assuming those are in seconds, your browser opens are more then 2 min for HDD and 33+ seconds for SDD
However, your text above the results implies you are shutting down sys-whonix between opens and forcing sys-whonix to restart each test. I would expect this to slow my 25 second results down.
I would expect my system to be faster then yours, so it sounds like i need to investigate the configuration i’ve been using to set systems up.
OK, I see. In my case (i7-3840QM, 16GB RAM, SSD) …
- 8.3 seconds for the debian-11-minimal procedure
- ~19 seconds for google-chrome to appear when launching a new disp vm (debian-11-minimal based)
I can see how with whonix this might take a bit longer, so I suppose this is a whonix related thing.
some context
I can also see how on first look this sounds like a big deal: 19 seconds to start the browser! … that is until one realizes that an entire virtual machine with 1GB RAM in my case got created, launched and the browser launched too. This and that as a result my visiting any website now is safer than it could ever be on a bare metal install.
I am not saying that there is anything wrong with wanting faster hardware and better performance, just trying to keep it all in perspective.
2 Likes
I don’t see a need to append anything since the test is clearly labeled as ‘very crude’. There’s no need to complicate things by referencing or adding boot times including apps and apps on heavier distros (Whonix), as what we have should give a good enough indication of real-world performance. If someone wants to make a Qubes performance benchmark that does include this data, it would make me (and I imagine Sven) happy, but this very crude test isn’t the place.
Also, I know that older machines are popular among the really hardcore infosec enthusiasts, mainly due to perceptions about ME vulnerabilities (why the AMD counterpart (PSP) is so often overlooked is beyond me). However, I think they are a vocal minority among Qubes users, if the hardware surveyed here and in the HCL is of any indication. Over time they’ll inevitably become an even smaller minority. I’d wager that for most the gaps between a VM launching and its app launching is usually small enough.
I noticed the test is heavily influenced by the initial memory setting of the template.
I tried increasing the initial memory to 16 GB which added more than 10 sec to the start time, but even small changes can change the start time by 1-2 sec. The lower the memory, the faster the template will start.
1 Like
I would test this observation but I’m not free at the moment. Hopefully someone will post some numbers and if they’re consistent enough I’ll consider adding another dimension to this test.
I wonder if VM disk size also has an impact.
I noticed that recently I could not reproduce the numbers in my past posts. Boot time, in general, has increased by about one second without mem-balancing, and two seconds or so with mem-balancing. I’m not quite sure what’s the cause, but the phenomenon inspired me that this thread can also be used for tracking performance across versions of infrastructural software (xen, libvirt, etc.), more or less helping performance improvement.
1 Like
I booted up my debian-11-minimal a few times and it came back as 10 seconds each, compared to 6.1 last time I checked. This was without shutting everything down though, so this is far from conclusive. I’ll do another round of tests once I find time.
Generally speaking I’ve found Qubes less stable and more sluggish than before.
Augsch, if you want to add a few more columns to the table, feel free to–but be sure to mark them as optional.
1 Like
Did actual tests and the result came back as around 6.7s, which is nowhere as slow as when everything was running, but 10% slower than earlier. Updated wiki.
1 Like
[added my measurements]
I’m using the performance governor (as hinted by @renehoj in this post) with 6 dom0 vcpus, pinned to E cores. Using the ‘ondemand’ governor is a tad slower (+0.5sec).
5.6s startup time is great compared to my old T450s that’s been running qubes for the past 7 years, but it’s 50+% more than @renehoj (which I assume ran the test on an MSI Z690-A Pro DDR4 - same as mine). I doubt it’s only down to the CPU (i9 12th gen vs i5 13th gen), it might be that btrfs also gives an edge with lvm operations when starting the vm.
[edit: forgot to add that disabling memory balancing as per @augsch 's advice shelves ~0.5s; total=5.1s]
Try enabling balancing, and lowering the max and init memory. I was playing around with the memory settings, they might have been lower than default.
1 Like
Disabling memory balancing does more for me. I can achieve an average of 3.2s with memory balancing turned off. However, I think the OP’s test scenario should be with memory balancing turned on.
1 Like
I’m fine with variations as long as they’re clearly marked so people can compare, like what @51lieal did with SSD-4k.
In fact, it’s nice to see this topic slowly morph into a thread on how to min-max qube startup times. I think as Qubes evolves, this metric will become increasingly important, since the logical conclusion of its compartmentalization-via-VM approach to security is to have a fresh VM for every little task–so how quickly a computer can spin up a new instance will be key. Even without this extreme, the usability of Qubes (as your daily driver at least) is strongly connected to how fast your computer can spin up VMs.
So a big thank you to all who have contributed so far–please keep at it! I’d contribute if I had anything to give, but this isn’t my field
3 Likes
I got interesting results running R4.2 Alpha on BTRFS and 4kn drive. The installer of R4.2 has new enough cryptsetup that can automatically select appropriate sector size. So I just need to switch my NVMe drive to 4kn mode ( using another LBA format sector size ).
So, booting debian-11-minimal without memory balancing, with its memory set to 400MB, takes ~2.5s. This is better than any previous results with different NVMe drives on my machine. Specifically, it’s ~0.6s faster than another BTRFS R4.2, which is installed on an identical drive, but without 4kn.
Now, the time between me clicking on the application launcher and chromium’s window showing up, is less than 5 seconds. And my laptop’s specs are far from “powerful”!
So for anyone interested in testing R4.2 and BTRFS, enabling 4kn mode perhaps will speed up your VMs booting!
There will be BTRFS optimization landing in kernel 6.3, and I’m hoping for another leap in performance.
2 Likes
try also 4kn template, i made the guide somewhere here, it should speed up vm by some 
also since this fresh install, can you try 4kn drive with lvm ? is it still not fixed ?
i also found that amd is way faster than intel (boot speed), my current i7-10750h vs ryzen 5 5600h.
i havent do any benchmark
2 Likes
Thanks for your guide! I’d love to try 4kn+lvm, but I just restored a ton of VMs ( to fully transit from 4.1 to 4.2, to enjoy the lightning speed 
), so maybe I’ll try 4kn+lvm when I get a spare nvme drive.
Anyway, my previous install was lvm without 4kn, and VM boot time is 1.5 seconds longer than my install of btrfs without 4kn. So I doubt whether 4kn+lvm will have performance gains conpared to 4kn+brtfs.
I’ll build 4kn templates to test if they will boot even faster. I read somewhere that btrfs treats the underlying fs’s sector size differently from lvm does. Given that 4kn templates do help in booting faster on lvm ( from your posts ) , it’s really worth investigating whether 4kn templates will boot faster than legacy templates, on brtfs.
I did some benchmarks on my main laptop before I converted it from tLVM to Btrfs-on-4k-luks. They are crude in that the only startup time measured is for Debian VM start + Firefox. However, I did also record VM shutdown times, and my experience is these really affect system performance when you are busily working on your system… the tLVM snapshot operations at shutdown hammer the dom0 storage layer often for 7+ seconds for medium sized volumes.
My plan is to re-do the benchmarks before long, so the restored VM volumes are still similar enough to make a fair comparison.
2 Likes
@tasket Great to see you again! Sorry for (temporarily) hijacking the thread, but I’m a long-time user of the tools you’ve written for Qubes–e.g. halt-vm-by-window, system-stats-xen, and more importantly, vm-boot-protect. I recommend the various scripts in this Github repo to just about anyone looking to have a more fluid Qubes experience–especially halt-vm-by-window.
Thank you very much for making tools that I use tens–maybe hundreds of times a day.
Anyways, the reason I’m reaching out is because I’ve noticed that since R4.1, vm-boot-protect tends to fail when booting up. I know I’m not supposed to use it with disposables, but I have since R4.0 without issue… until R4.1. Now, there’s a high probability that a terminal window appears with a warning that vm-boot-protect was triggered and that the bad private files are in /mnt/xdvb (not accurate; recalling from memory).
I’d hugely appreciate it if you updated your tools for R4.1 (or even R4.2).
1 Like
With the work you and others are doing, a sub-1-second boot seems to be a fast-approaching reality. Hopefully with some template-side-tweaks (or even a native specialized template like debian-15-quick) this should be attainable within a few years.
Maybe I should start a thread asking people for ideas about what could be done with this–i.e. how to leverage the almost-negligible cost of booting (and shutting down) a VM to make Qubes more secure and more usable.
1 Like