Disappointment in speed of Xen in Qubes

[Crafts]

Speed Comparison Xen, KVM and VirtualBox

Quote from a variety of popular sources:

Based on general performance trends for these virtualization technologies, here’s how they compare to the native system for a typical mixed workload (combining CPU and I/O operations):

Xen: Xen runs at approximately 98% of the native system’s speed. This means it is about 2% slower due to the minimal overhead introduced by its virtualization layer. Its efficiency stems from running directly on the hardware and supporting features like paravirtualization for optimized I/O.
KVM: KVM operates at about 99% of the native system’s speed, making it approximately 1% slower than the native system. Its slight edge over Xen comes from its tight integration with the Linux kernel and the use of virtio drivers for efficient I/O handling, bringing it very close to native performance.
VirtualBox: VirtualBox performs at around 95% of the native system’s speed, meaning it is approximately 5% slower than the native system. As a type 2 hypervisor, it incurs additional overhead because it runs on top of a host OS, though hardware virtualization extensions and paravirtualization options help mitigate this to some extent.

Key Notes

These percentages are approximate and reflect general performance for a balanced workload. Actual speeds can vary depending on:
    Workload Type: CPU-intensive tasks may show all three hypervisors closer to 99% of native speed, while I/O-intensive tasks might widen the gap, especially for VirtualBox (potentially dropping to 90% or lower).
    Configuration: Optimizations like enabling hardware virtualization (e.g., Intel VT-x or AMD-V) or using paravirtualization can improve performance.
KVM tends to have the least overhead, followed closely by Xen, with VirtualBox trailing slightly due to its type 2 nature.

Summary

In relation to the native system:

Xen: 98% (2% slower)
KVM: 99% (1% slower)
VirtualBox: 95% (5% slower)

This comparison provides a clear picture of their relative performance, with KVM being the closest to native speed, Xen very slightly behind, and VirtualBox a bit further due to its additional layer of abstraction.

I decided to test this data for Qubes running Xen using the ‘7z b’ test.

First, I installed a clean Debian on my computer and measured the speed.
It turned out to be 16929 MIPS:

Let’s take it as 100%.

Then I installed VirtualBox, virtual Debian in it and measured the speed again.
It turned out to be 11268 MIPS:

VirtualBox, this is 67%, which is much less than the promised 95%.

After that I installed Qubes on the same computer and measured the speed in Debian AppVM.
The speed was 11895 MIPS:

Xen on Qubes. this speed is 70% of the natural speed, i.e. 30% less than the expected speed of Xen.

[Crafts]

[Crafts]

This speed is 70% of the natural speed.

Why are Xen speeds in Qubes is it significantly lower than the expected 98% speed?

[Crafts]

But, the situation with Xen turned out to be even worse!
The previous Virtualbox test showed a speed of 67%, but only 4 cores were used here.

And now I’m using all 8 cores:

Now Speed VirtualBox = 15012 MIPS, this is 89%.

Total speed of:
VirtualBox = 89%
Xen in Qubes = 70%

So, Qubes &Xen shamefully lost even to VirtualBox??

[solene]

What is your CPU?

If it has E and P cores, Xen does not know how to handle it and will not care. Maybe the kernel of the OS where virtual is running will spread the load on P cores instead of using random cores.

[Crafts]

I hope you will agree with me that it is not my task to solve the puzzle of cores.
Xen has to do this on its own, and I don’t even need to know how it does it.
This is exactly what VirtualBox does, and it outperforms Xen under the same conditions with the same CPU.

[OvalZero]

I hope you will agree with me that synthetic benchmarks don’t describe things more accurately than marketing papers.

Lies, Damned Lies and Benchmarks

What is your intended use?
What hardware do you have?

[Crafts]

…synthetic benchmarks don’t describe things more accurately than marketing papers

Synthetic tests under the same conditions make it possible to compare results well in relative units.

@OvalZero

What hardware do you have?

Intel i7 4th generation
RAM 16 GB
SSD 128 GB

@OvalZero

What is your intended use?

I don’t know yet, I’m just learning Qubes.
And if the speed Qubes does not lose too much to the native speed, then I plan to replace Debian with Qubes for everyday diverse work with increased my data security.

Because I very like the concept of Qubes

[renehoj]

Did you enable smt?, the default Qubes OS settings is running Xen with smt=off.

[Crafts]

No, I didn’t turn it on. Is the default value off wrong?

[renehoj]

Off is the correct setting, but it is picked for security, not performance.

Disabling smt comes with a noticeable performance penalty, but it’s the only way to protect the system against transient execution vulnerabilities.

In artificial benchmarks, you could see 20% performance difference between smt on or off. If you are comparing Xen against a hypervisor that is using smt, you should enable it for Xen, if you want an apples to apples comparison.

[OvalZero]

[renehoj reacted more quickly.]

[Crafts]

Well, I tried to do thorough testing of Qubes and got this speed consistent result : 70%.

Could someone kindly do the same test?
It’s simple and won’t take much time, but it will make sure that my Qubes runs at the same speed as yours.

[phceac]

If you want to compare like with like, then it will be necessary to manage the testing method a bit. It looks like the test scores assume 100% of resources are fully available.

Your VM is being told it has 8 vCPUs, but that doesn’t mean 7z is immediately getting 99.9% access to 8 physical CPU cores as it does in plain Debian. It is visible in the usage column… especially for the 8 and 16 Mb qubes tests which show a clear problem.
Maybe xentop can indicate what else is using the processors, but 0% CPU utilisation for a test is just weird. It is also not clear why the compression test can only use <6 cores, even on bare hardware. These look like artefacts of the test.
[Édit: I am assuming there is major contention for the CPUs… my experience of thrashing hardware is that contention is bad for performance, although I think that was trying multiple VMs running the Phoenix workbench]

Similarly, the RAM usage columns seem quite different between the tests. Qubes is telling the VM it has 4 GB, not all 16, like in Debian. By default it would not immediately allocate 4 GB, and any overhead of qmemman would (might?) directly affect a single run of the test.

At least, it is would be logical to:

1. ensure that other VMs are shut down (but be careful if mouse and keyboard are on sys-usb, because you will lose control if you shut it down).
2. Make sure your test VM has enough initial memory allocation, with balancing disabled.
3. Run repeats, either by multiple runs of the single test, or using the argument of 7z.
4. (and make sure smt is off in Debian)

Maybe there is more you could do to make the tests comparable. Even then, this test is not really representative of normal use - depending on what is normal for you. Ultimately, there will be some price to pay for all that Qubesy goodness, but it seems unlikely it would be 30% of 8 cores. It is more probable that you are seeing the effects of Qubes keeping resources where they are most useful, until a VM shows a need.

[Crafts]

Yes, I agree that the other AppVM should be turned off, and they really were turned off.
The rest of what you propose means creating special conditions for testing Qubes But no one does this in normal operation, and VB demonstrates high speed even without creating special conditions.

If, in your opinion, 4 GB of allocated memory affects the measurement results, then I can increase it.
But in my humble opinion, for 7zip, it won’t affect the test results.

[phceac]

@Crafts

special conditions for testing Qubes

Maybe it’s Debian and Vbox which have been given special conditions… no Dom0 with qubes infrastructure, no memory ballooning unless it is now Vbox default, no graphics isolation, etc… plus a residual uncertainty about whether Linux could be re-enabling SMT.

If you want to test CPU under xen, then all those other effects need eliminating, one way or another.
The initial provisioning of extra memory to a VM which starts with small memory under qubes can cause a short delay to a new process, which is normally imperceptible, but could play badly with a pure CPU benchmark. I’m not sure that is what’s happening in your test - I really don’t have an opinion - but I would want to eliminate it before saying there’s a general 30% overhead on the processor speed. There’s no value to a benchmark result which is dominated by a minor artefact… unless that specific artefact is what is important to you.

[Crafts]

Okay, if in your opinion the 7z utility is not well suited for speed testing of Qubes, then what more objective means can you suggest for this?

[phceac]

@Crafts

7z utility is not well suited

This is not my opinion - I have not looked at the 7z benchmark beyond a quick scan of the man page, and seeing a per-processor performance of more than 33000000000 MIPS on one of the screen captures. I’m pretty sure that value is wrong, but it is not even clear to me what the final scores represent.

My limited experience is that virtualization makes it somewhat easy to obtain worse performance without the cause being visible from inside the VMs. My only use of performance benchmarks has been for testing hardware reliability, so I have tended to use platforms which can load a wider range of components, such as Phoronix(& 1 other, ages ago). It’s another thing to determine whether it can give meaningful information about performance.

[Atrate]

@Crafts

Quote from a variety of popular sources:

What sources? Please cite them. From the sentence structure of your “quoted” post it looks very much like generated by an LLM…

[Crafts]

Absolutely right. At first, I manually studied various disparate sources on the Internet. And then, for the convenience of presenting the material, I asked the LLM to speak.
The result of the sources and LLM matched.