Xen -> KVM nested virtualization for testing

General Discussion
1

Hey @marmarek

Can you give pointers to current bugs and qubes tickets to follow explaining why Xen is broken and not support this?

I’m testing Heads development on top of qemu under a standard qube and as you know, qemu tcg is used since KVM cannot.

I would love to know and be able to tackle the issues upstream and tracking current state would be helpful.

Feel free to move subject to proper category

1 Like
2

I believe it was because of it increase greatly attack surface for dom0.

I read it somewhere in the forum someone can install proxmox and rum vm on it.

I can confirm it works, but not sure about stability.

3

Totally get it. Ideally, Xen nested virt should stay as a testing version where user could select at boot if nested virt is desired.

I’m such user who would need it to develop things.

Don’t get me wrong, qemu tcg works. But IO are so slow it reduces testing speed so much I am considering using another laptop to do development and testing. This is sad.

Ideal would be either

  • have a switch to permit Xen to be booted with nested virt explicitly (Heads could have configuration setting toggle for that, turning Qubes in developer mode)
  • Have qubes add a grub entry for Xen supporting nested virt and have the user select that boot option, where Xen nested virt would need to be installed from testing repo and never land under normal repositories.

@marmarek : thoughts?
Pointers to Xen discussions/qubes issues would be helpful

4

Does this help?

5

@adw yes and no. I understand that Xen → Xen nested virt might be totally wrong. But I would love direct links on what are the actual problems into having KVM working under Xen. Totally broken doesn’t leave practical points that need to be perfected to be able to use Qubes in other way then qemu (TCG) under it to run test and if that is the conclusion, I would love to understand why.

The issue linked is closed for discussion so what ticket should be opened to be able to discuss the minimal things broken inside of xen to be able to run kvm under it?

@adw: there is a difference between

  • Expecting Xen to be able to run under other virtualization solutions
  • Have Xen support KVM by having Xen support nested virtualization.

This discussion is aimed to support the second option: that is, being able to run kvm under a qube. What works, what doesn’t, where we are at and we can and cannot go. What is broken, what can be fixed and not.

Some traces

Some not so relevant bing chat output:

I see. So you want to run KVM inside a Qube, which is a virtual machine managed by Xen and Qubes OS. You are currently using QEMU TCG, which is a software emulator that does not use hardware acceleration, and you want to switch to KVM, which is a hypervisor that uses hardware acceleration¹.

I did some web search and found some relevant discussions on this topic. It seems that there are some blockers that prevent you from running KVM inside a Qube, such as:

  • Qubes OS does not support nested virtualization, which is the ability to run a hypervisor inside a virtual machine. Nested virtualization is considered both experimental in Xen and a potential security liability in Qubes OS².
  • Qubes OS requires direct access to the hardware for security and performance reasons, and does not allow passing through devices such as GPUs to the Qubes³ . This may limit the functionality of KVM and the guest VMs running on it.
  • QEMU TCG and KVM use different execution modes and have different requirements for the guest VMs. Switching between them may not be possible or easy without modifying the QEMU code or the guest VM configuration.

I’m afraid I don’t have enough knowledge or authority to provide you with a working plan to fix these issues. You may need to contact the developers or maintainers of Qubes OS, Xen, or KVM for more information and guidance. Alternatively, you may want to consider other solutions for your use case, such as using another hypervisor or another operating system that supports nested virtualization.

I hope this helps you understand some of the challenges of running KVM inside a Qube. If you have any more questions, please feel free to ask me. :blush:

Source: Conversation with Bing, 9/5/2023
(1) Does QubesOS support running a QEMU/KVM VM within a Qube?. Does QubesOS support running a QEMU/KVM VM within a Qube?.
(2) Can QEMU switch between KVM and TCG dynamically?. Can QEMU switch between KVM and TCG dynamically? - Stack Overflow.
(3) Documentation/TCG - QEMU. Documentation/TCG - QEMU.

6

Following that rabbit hole:

I guess my question goes back here to @marmarek.
If I was to follow that path one year later, with Xen 4.17 to expose vmx and svm to trusted testing qube where I would be responsible for trusting it with what I’m testing.
I expect to not be lucky because of Xen upstream bugs or qubes building options being turned off.

What would be the proper path into bringing those for people needing it.

Background of need felt by the community with opened threads and issues

  • Android studio needs KVM. I ran into that doing a vulnerability report. And needed another machine to create poc code and be able to run it into Android virtual machine to have a proper workflow. I do not see, in practice why this would be a problem for qube. That android testing VM should be trusted/untrusted accordingly.
  • Heads now provides qemu/KVM q35 support for testing and development. When those board are run under Qubes, KVM mode falls back to qemu tcg mode, where installing an OS takes several hours because of IO being emulated instead of virtualized through virtio and KVM kernel support, which when run baremetal is basically near native performance. If unavailable this means my workflow needs to run on baremetal otherwise only basic testing can occur under a qube, making things harder to test/develop.

Those two workflows alone justify to state the actual limitations of permitting, or deploying a Xen-nested additional version under qubes-testing or even qubes-unstable for users to be able to use KVM.

Again: not the other way around: the need is to have Xen support KVM. Not to have Xen (or Qubes) run under nested virtualization.

7

Did you read this post about running Proxmox?

8

Unlocked, but I encourage everyone to avoid using qubes-issues for discussion if at all possible:

https://www.qubes-os.org/doc/issue-tracking/#the-issue-tracker-is-not-a-discussion-forum

9

I stopped at “constantly crashes”

10

Kindly bumping.

OP is raising some very valid points. It is saddening that this cannot at least be addressed to a point where the feature can be testable in a strictly contained environment.

I will also be more than glad to contribute to what is preventing HVM support. Qubes running under nested visualization is not the issue, and the absurdity of it is not worth discussing. But KVM support is becoming increasingly relevant as an android developer since the Eclair era.

1 Like
11

Suggests original post participants were somewhat successful doing so.
Referring to this post in matrix official channel

Github issue : unlocked/closed with not planned tag:

Cross-posted to official qubes-public channel You're invited to talk on Matrix

12

To track progress/past discussions upstream Nested Virtualization on Xen Revamp (&25) · Epics · xen-project · GitLab

13

The current state of the nested virtualization in Xen and further development:

1 Like