Installing Qubes 4.1 in a Xen HVM domU (nestedhvm=1)

User Support General
1

Hello, has it been tested ? If yes what are the gotchas ?
I don’t know if it’s related, but using 2 sparse and empty file-backed disks (disk0.img 40GiB, disk-swap.img 4GiB), it seems the automatic partitioning does not work, when i select the 40GiB disk, i get “new lv is too large to fit in free space”.
Meanwhile waiting for answers, will try manual partitioning and report back !

PS/edit: both “automatic part” and “custom->create automatically” fail btw

1 Like
2

Even up to the current version of Xen, Nested Virtualization inside Xen is considered either Tech Preview or Experimental (for PV and HVM respectively, both inside HVM).

As far as I am aware, no one has gotten reliable results using it within Qubes.

https://xenbits.xen.org/docs/unstable/support-matrix.html
https://xenbits.xen.org/docs/unstable/SUPPORT.html#x86nested-pv

Brendan

2 Likes
3

Ah damn, I knew that was experimental, but thanks for your insights about Qubes.
I just knew Xen devs were using it to test Xen inside Xen, so I was like, let’s go and try.
By the way, are Qubes domU in PV, HVM or PVHVM mode ? Is it selectable ? i don’t remember if Qubes uses Xen PV drivers or MUST use them.

1 Like
4

dom0 is PV
domU can be HVM or PVH.

While PV domU is also supported in 4.0 under Xen 4.8 (but not recommended), that might not work in 4.1. I don’t have access to my 4.1 test box at the moment, but there’s an open issue for PV on 4.1 here: Linux PV domain crashes on Xen 4.14 · Issue #6052 · QubesOS/qubes-issues · GitHub .

B

5

If there wasn’t a PV mode, a lot of USB controllers couldn’t/wouldn’t work/be used in Qubes via sys-usb…

6

Sys-usb in 4.0/4.1 is HVM.

And as far as I’m aware, under 4.0 and 4.1, VM’s that utilize sys-usb’s shared devices are PVH (aka PVHVM), not PV.

Brendan

7

I meant on something like this.

8

Ah, ok. Right.

With R4.0 we saw use of PV mode being deprecated for IOMMU-less systems, but the installer allowed installing without IOMMU, and PV mode was the only way to passthrough PCI (less-safely).

However, R4.1 now requires a functioning IOMMU to install.

Are there that many USB controllers on systems with a functioning IOMMU that are incompatible with the IOMMU?**

B

** I shouldn’t be surprised if the answer is yes, I suppose.

9

However, R4.1 now requires a functioning IOMMU to install.

Not quite true, the hard dependency was removed somewhere in the alpha iterations, you can proceed without IOMMU (the installer just show a warning)

However, sys-net / sys-usb will fail to launch as HVM without IOMMU, and if you try to change to PV, it will also fail to launch, because you need to explicitly enable this kernel flag: qubes.enable_insecure_pv_passthrough (commit)

So besides the less security of PV, for system without IOMMU at least they can enjoy other Qubes features.

1 Like
10

Thanks all for the links and information ! I’ll see what works best for me ^^ For now I don’t really need PCI PT as I’m using X on dom0. I don’t know if nested PCI PT is even possible. I’d only need it for audio, but why is audio handled by a separate domain in Qubes, security ? For now I use soundhw="hda" in domU config and pulseaudio/pax11publish to play in dom0.

Anyways, I finally managed to install and run Qubes 4.0.4 in a HVM domU, and I have questions :

  • how does the IOMMU work in case of nested virt ? I -think- I have one in dom0 (iommu: Default domain type: Translated [\n] AMD-Vi: AMD IOMMUv2 functionality not available on this system), but Qubes install reports I don’t have it. ls /sys/kernel/iommu_groups is empty on dom0.
  • is “hap = 1” needed ? I don’t know what it does really, but xen wiki/manpages tell to use it with nested virt. I’ve not seen (yet ?) any difference with or without, I even think it’s at 1 by default anyways. Although since I activated it in qubes.cfg, I have many lines like this in xenstored.log: “data/meminfo_free 2405480 [\n] control/feature-balloon 1”, don’t know if it’s related or not. Autoballooning is disabled in my dom0 (and by default in qubes’), and I use the same values for dom0_mem and max in both xen cmd lines. Well, I changed it in qubes only AFTER the first boot.
  • when qubes boots, I get a few “(XEN) d16v0: Invalid EFER update: 0x1d01 -> 0x3901 - Unknown bits set” lines, what does that mean ?
  • when qubes run, the dom0 xl dmesg log is filling with “(XEN) d53v1 Unexpected nested vmexit: reason 0x6e” and “[...] reason 0x87”, to the point that the command never stops ! Where can I read what those error messages mean ? And how can I increase xl dmesg size ? I lost all previous pertinent logs …
  • should I run qubes domU in HVM or PVH mode ?
  • do you know someone who tests nested virt ? ^^
  • off-topic, but all my attempts at installing qubes from a NFS-backed ISO failed with “Payload SHA256 digest: BAD” on various packages, any idea why ?

And please, don’t stop discussing, I learn a lot ; )

1 Like
11
12

I see. Last i thought I read, it was the intention to remove it in R4.1, appears not to be the case though.

B

13

Based on my tests I can confirm the warning is the same between 4.0.4 and 4.1.0, it’s just a warning and one can continue the installation, which I find is the correct way to go (more adoption) !
Note that I still can’t install 4.1.0 cause of a luks problem, even if at some point I succeeded but with another (now solved) problem, unfortunately I don’t remember how … So for now I only test 4.0.4.

Now, no one for any of my questions ? ^^
All questions are still valid, but here are some updates (the numbering reflects the order in my previous post) :

    1. the note about data/meminfo_free 2405480 [\n] control/feature-balloon 1 is irrelevant, I checked the logs and it happens all the time. But the initial point is still valid: is hap needed
    1. I still don’t know what the vmexit errors mean, but I found a way to log xl dmesg in text files. In the Xen source code, the xl dmesg ring buffer size is hardcoded to 16k. The solution is to set XENCONSOLED_TRACE in /etc/default/xen (that would be sysconfig in Qubes), as per this post, but I have yet to make it work, seems the SIGHUP is not enough.

The vmexit thing seems the most annoying for now. But Qubes domUs seem to work, even if I only have sys-net and sys-fw running.

14

Ok, “Qubes dom0 as a domU” v4.1 installed and running ! I could even start the sys-net domU, but only in PV mode (HVM = 100% CPU usage and PVH = crash on boot).
I also realized the vmexit log lines filling my dom0 xl dmesg were caused by using a qubes domU in HVM mode. In PV mode, all good !
Also, for point number 2 about hap=1, the answer was on the “nested virt” page of the Xen wiki …
For now the requirements I found mandatory to have a nested dom0 working:

# builder="hvm" # deprecated
type="hvm"
nesthvm=1
hap=1
# using PV drivers notation for disks is ok (maybe because it falls back to hda ? didn't check yet) ...
disk = [ 	'qubes4.1/disk0.img,,xvda' ,
		',,xvdb,cdrom' ,
		]
# ... but not for the network, the "vif" is not detected by qubes dom0, but the "e1000" is
vif = [	'bridge=xbr-pftests, mac=00:16:3e:48:40:40, vifname=qubes41,model=e1000' ,
		'bridge=xbr-pftests, mac=00:16:3e:48:40:41, vifname=qubes41-pv,type=vif' ,
	]

So I’m left with:

  • (old 1.) trying to see if I have a IOMMU and if yes get it to work (BIOS and/or microcode update ?)
  • (old 3.) Invalid EFER update, I have as many lines as I have vCPU given to qubes
  • this bug (qubes not bootable cause 100% CPU), but not sure if it’s related to a nested virt prob or another, no log avail
  • when from Qubes dom0 I do xl console DOMAIN (as I can’t terminal in yet …), the console is really awful (lots of control characters printed), why is that ?
15

I always tend to ask myself why, instead of how, and when instead of if, so while this sounds interesting, why we would want to run Qubes in domU under Xen as dom0?

16

Ahah, yeah nice way to think, I agree ^^ Even if IT, for me, is not always “I need this”, but “I want to test this” ! When you’re growing up, only the toys change, not the mentality ^^
Anyways, there are many reasons ! Some are related to my specific use case, some may be more general.

Specific to me :

  • First and foremost, because I already have a dom0 host, and I want to try Qubes ! Also I don’t have another capable and powerful enough machine to run Qubes.
  • As explained in this post (read only the 2 first §), my dom0 hosts domUs/services used by my local network.
  • I can “port” Qubes functionalities into my system to render it more usable and secure, as well as learning new management tricks about Xen and systems management.
  • Why not ? ^^ I like tinkering and discovering new things !

More genericly :

  • people often try new OS in virtual machines, so I thought people could benefit from my tests. But ok, I admit Xen is not the typical hypervisor user choice !
  • related, testing Qubes from Qubes may be useful to Qubes developpers and enthousiasts alike, to test new kernels, functionnalities, etc, without breaking the main install (even though not everything could be validated from a Qubes domU).
  • I’ve read Qubes is thinking about the cloud (but if I understood correctly, not for dom0 rather the domUs), and some providers use Xen as their hypervisor, so it would allow to test/run a full Qubes dom0 there. Some users could use Tails/Whonix on local hardware to connect to a cloud-based Qubes dom0.
  • testing nested virt provides the Xen project more information/debug about this experimental feature.

And this one is a mix of both reasons:

  • Qubes is a “user oriented” system, so it makes sense to run it virtualized on a more “infrastructure oriented” system (like mine), even if I agree that it poses security risks that Qubes may not be able to handle (but should try to ?). Even if Qubes’ main goal looks to be the ultimate most secure OS (why not openBSD-based btw ?), it is already used by people who don’t care about EVERY security aspects of it. And to me having more users is a good thing for a distro, so the more use cases Qubes cover, the merrier ! With all due warnings of course.

There are maybe things I dont think of right now, but I guess it’s enough good reasons, no ?
Sorry long post, but you asked … ^^

PS to moderators: could our 2 last posts be split into another discussion in “General discussion” please ? It would be more relevant than here (User support 4.1) and I’d like to keep this post related to the technical aspects of it. Or move the whole conversation, and I’ll create new posts about it ? As you want, TIA !

2 Likes
17

Thanks for your response, I find it very useful for all that might read it, and I must say that somehow I felt that “temptation to make it run as Domu” was one of the main reasons. While until qubes can be run only in PV mode I don’t see it’s full potential can be used, still more than interested to follow your findings. Thanks for being first to write, if not to test, this.
Let’s flag the topic, that should be the fastest way to summon mods. :slight_smile:

1 Like
18

Thanks for the nice comments, I hope it attracts more attention as I think the use cases are interesting, and not only for my little person ! ^^

Thanks, but first, oh no ^^ Look those posts, from 2015/2016 !

This is totally uncharted territory for me, dunno what you mean ^^

2 Likes
19

… to call the mods to split the topic… :slight_smile:

20

Ah, you mean I flag my own post with a message to ask for splitting, so I can create a new topic which could be titled “Why running Qubes as a domU from another dom0 can be useful” ?