Installing Qubes 4.1 in a Xen HVM domU (nestedhvm=1)

User Support General
6

Sys-usb in 4.0/4.1 is HVM.

And as far as I’m aware, under 4.0 and 4.1, VM’s that utilize sys-usb’s shared devices are PVH (aka PVHVM), not PV.

Brendan

7

I meant on something like this.

8

Ah, ok. Right.

With R4.0 we saw use of PV mode being deprecated for IOMMU-less systems, but the installer allowed installing without IOMMU, and PV mode was the only way to passthrough PCI (less-safely).

However, R4.1 now requires a functioning IOMMU to install.

Are there that many USB controllers on systems with a functioning IOMMU that are incompatible with the IOMMU?**

B

** I shouldn’t be surprised if the answer is yes, I suppose.

9

However, R4.1 now requires a functioning IOMMU to install.

Not quite true, the hard dependency was removed somewhere in the alpha iterations, you can proceed without IOMMU (the installer just show a warning)

However, sys-net / sys-usb will fail to launch as HVM without IOMMU, and if you try to change to PV, it will also fail to launch, because you need to explicitly enable this kernel flag: qubes.enable_insecure_pv_passthrough (commit)

So besides the less security of PV, for system without IOMMU at least they can enjoy other Qubes features.

1 Like
10

Thanks all for the links and information ! I’ll see what works best for me ^^ For now I don’t really need PCI PT as I’m using X on dom0. I don’t know if nested PCI PT is even possible. I’d only need it for audio, but why is audio handled by a separate domain in Qubes, security ? For now I use soundhw="hda" in domU config and pulseaudio/pax11publish to play in dom0.

Anyways, I finally managed to install and run Qubes 4.0.4 in a HVM domU, and I have questions :

  • how does the IOMMU work in case of nested virt ? I -think- I have one in dom0 (iommu: Default domain type: Translated [\n] AMD-Vi: AMD IOMMUv2 functionality not available on this system), but Qubes install reports I don’t have it. ls /sys/kernel/iommu_groups is empty on dom0.
  • is “hap = 1” needed ? I don’t know what it does really, but xen wiki/manpages tell to use it with nested virt. I’ve not seen (yet ?) any difference with or without, I even think it’s at 1 by default anyways. Although since I activated it in qubes.cfg, I have many lines like this in xenstored.log: “data/meminfo_free 2405480 [\n] control/feature-balloon 1”, don’t know if it’s related or not. Autoballooning is disabled in my dom0 (and by default in qubes’), and I use the same values for dom0_mem and max in both xen cmd lines. Well, I changed it in qubes only AFTER the first boot.
  • when qubes boots, I get a few “(XEN) d16v0: Invalid EFER update: 0x1d01 -> 0x3901 - Unknown bits set” lines, what does that mean ?
  • when qubes run, the dom0 xl dmesg log is filling with “(XEN) d53v1 Unexpected nested vmexit: reason 0x6e” and “[...] reason 0x87”, to the point that the command never stops ! Where can I read what those error messages mean ? And how can I increase xl dmesg size ? I lost all previous pertinent logs …
  • should I run qubes domU in HVM or PVH mode ?
  • do you know someone who tests nested virt ? ^^
  • off-topic, but all my attempts at installing qubes from a NFS-backed ISO failed with “Payload SHA256 digest: BAD” on various packages, any idea why ?

And please, don’t stop discussing, I learn a lot ; )

1 Like
11
12

I see. Last i thought I read, it was the intention to remove it in R4.1, appears not to be the case though.

B

13

Based on my tests I can confirm the warning is the same between 4.0.4 and 4.1.0, it’s just a warning and one can continue the installation, which I find is the correct way to go (more adoption) !
Note that I still can’t install 4.1.0 cause of a luks problem, even if at some point I succeeded but with another (now solved) problem, unfortunately I don’t remember how … So for now I only test 4.0.4.

Now, no one for any of my questions ? ^^
All questions are still valid, but here are some updates (the numbering reflects the order in my previous post) :

    1. the note about data/meminfo_free 2405480 [\n] control/feature-balloon 1 is irrelevant, I checked the logs and it happens all the time. But the initial point is still valid: is hap needed
    1. I still don’t know what the vmexit errors mean, but I found a way to log xl dmesg in text files. In the Xen source code, the xl dmesg ring buffer size is hardcoded to 16k. The solution is to set XENCONSOLED_TRACE in /etc/default/xen (that would be sysconfig in Qubes), as per this post, but I have yet to make it work, seems the SIGHUP is not enough.

The vmexit thing seems the most annoying for now. But Qubes domUs seem to work, even if I only have sys-net and sys-fw running.

14

Ok, “Qubes dom0 as a domU” v4.1 installed and running ! I could even start the sys-net domU, but only in PV mode (HVM = 100% CPU usage and PVH = crash on boot).
I also realized the vmexit log lines filling my dom0 xl dmesg were caused by using a qubes domU in HVM mode. In PV mode, all good !
Also, for point number 2 about hap=1, the answer was on the “nested virt” page of the Xen wiki …
For now the requirements I found mandatory to have a nested dom0 working:

# builder="hvm" # deprecated
type="hvm"
nesthvm=1
hap=1
# using PV drivers notation for disks is ok (maybe because it falls back to hda ? didn't check yet) ...
disk = [ 	'qubes4.1/disk0.img,,xvda' ,
		',,xvdb,cdrom' ,
		]
# ... but not for the network, the "vif" is not detected by qubes dom0, but the "e1000" is
vif = [	'bridge=xbr-pftests, mac=00:16:3e:48:40:40, vifname=qubes41,model=e1000' ,
		'bridge=xbr-pftests, mac=00:16:3e:48:40:41, vifname=qubes41-pv,type=vif' ,
	]

So I’m left with:

  • (old 1.) trying to see if I have a IOMMU and if yes get it to work (BIOS and/or microcode update ?)
  • (old 3.) Invalid EFER update, I have as many lines as I have vCPU given to qubes
  • this bug (qubes not bootable cause 100% CPU), but not sure if it’s related to a nested virt prob or another, no log avail
  • when from Qubes dom0 I do xl console DOMAIN (as I can’t terminal in yet …), the console is really awful (lots of control characters printed), why is that ?
15

I always tend to ask myself why, instead of how, and when instead of if, so while this sounds interesting, why we would want to run Qubes in domU under Xen as dom0?

16

Ahah, yeah nice way to think, I agree ^^ Even if IT, for me, is not always “I need this”, but “I want to test this” ! When you’re growing up, only the toys change, not the mentality ^^
Anyways, there are many reasons ! Some are related to my specific use case, some may be more general.

Specific to me :

  • First and foremost, because I already have a dom0 host, and I want to try Qubes ! Also I don’t have another capable and powerful enough machine to run Qubes.
  • As explained in this post (read only the 2 first §), my dom0 hosts domUs/services used by my local network.
  • I can “port” Qubes functionalities into my system to render it more usable and secure, as well as learning new management tricks about Xen and systems management.
  • Why not ? ^^ I like tinkering and discovering new things !

More genericly :

  • people often try new OS in virtual machines, so I thought people could benefit from my tests. But ok, I admit Xen is not the typical hypervisor user choice !
  • related, testing Qubes from Qubes may be useful to Qubes developpers and enthousiasts alike, to test new kernels, functionnalities, etc, without breaking the main install (even though not everything could be validated from a Qubes domU).
  • I’ve read Qubes is thinking about the cloud (but if I understood correctly, not for dom0 rather the domUs), and some providers use Xen as their hypervisor, so it would allow to test/run a full Qubes dom0 there. Some users could use Tails/Whonix on local hardware to connect to a cloud-based Qubes dom0.
  • testing nested virt provides the Xen project more information/debug about this experimental feature.

And this one is a mix of both reasons:

  • Qubes is a “user oriented” system, so it makes sense to run it virtualized on a more “infrastructure oriented” system (like mine), even if I agree that it poses security risks that Qubes may not be able to handle (but should try to ?). Even if Qubes’ main goal looks to be the ultimate most secure OS (why not openBSD-based btw ?), it is already used by people who don’t care about EVERY security aspects of it. And to me having more users is a good thing for a distro, so the more use cases Qubes cover, the merrier ! With all due warnings of course.

There are maybe things I dont think of right now, but I guess it’s enough good reasons, no ?
Sorry long post, but you asked … ^^

PS to moderators: could our 2 last posts be split into another discussion in “General discussion” please ? It would be more relevant than here (User support 4.1) and I’d like to keep this post related to the technical aspects of it. Or move the whole conversation, and I’ll create new posts about it ? As you want, TIA !

2 Likes
17

Thanks for your response, I find it very useful for all that might read it, and I must say that somehow I felt that “temptation to make it run as Domu” was one of the main reasons. While until qubes can be run only in PV mode I don’t see it’s full potential can be used, still more than interested to follow your findings. Thanks for being first to write, if not to test, this.
Let’s flag the topic, that should be the fastest way to summon mods. :slight_smile:

1 Like
18

Thanks for the nice comments, I hope it attracts more attention as I think the use cases are interesting, and not only for my little person ! ^^

Thanks, but first, oh no ^^ Look those posts, from 2015/2016 !

This is totally uncharted territory for me, dunno what you mean ^^

2 Likes
19

… to call the mods to split the topic… :slight_smile:

20

Ah, you mean I flag my own post with a message to ask for splitting, so I can create a new topic which could be titled “Why running Qubes as a domU from another dom0 can be useful” ?

21

Hahah, I flagged whole topic asking them to split the topic by your suggestion

22

Ah ah, ok, thanks then ^^ I don’t know how that works nowadays, but let’s see, you learn something new everyday (was the phpbb2 dev/admin for my gaming clan/friends) !

23

Sorry, I’ve been trying to split this, but I lack context and the discussion goes beyond my technical knowledge. So I’m afraid I can’t help much.

So it would be better if you create another thread in general discussion summarizing the discussion the discussion stemmed from here (for example via a linked topic).

Also, when requesting a topic split, a way to do it is to flag the topic as @enmus suggested, but also saying which posts you want to split and what’s the new topic name. Without that key context and especially on highly technical topics it can be hard for mods to know.

1 Like
24

Thanks. I’ll keep that in mind on occasions splitting would be my idea.

1 Like
25

So, I finally got a working nested Qubes ! I mean, connected to the network.
The trick is to use a USB Ethernet adapter, as USB passthrough works, on the contrary to PCI.
WARNING : this method is only for testing purposes !!!

Terminology

  • L0-dom0 : this is the dom0 installed on bare metal, a vanilla Xen on Debian for me, no libvirt.
  • L1-dom0 : this is the Qubes install, which is a HVM domU running on L0-dom0. I’ll refer to it as Qubes or Qubes dom0 or Qubes L1.
  • L2-domU : all the domUs started by Qubes (L1-)dom0. So sys-net, sys-firewall, etc

Problems/limits

  • only PV L2-domUs are supported. PVH and HVM crash Qubes L1-dom0 (vmexit errors reported in L0-dom0)
  • PCI passthrough-ing emulated hardware (like the QEMU e1000) from Qubes L1 to L2-domUs won’t work.

How-to steps

Except for non-PV L2-domUs, Qubes run fine nested, and works as expected.
The hard part is to get a working external network connection.
I solved it by using an USB Ethernet device. It is connected in L0-dom0, and USB passthrough-ed to Qubes L1.

There are 4 main steps :

  1. attach the USB device from L0-dom0 to Qubes L1
  2. allow the USB device through USBguard
  3. create a network stack in Qubes L1
  4. create a custom config file for L2 sys-net
(hack, sorry, didn't find an elegant way to reset a numbered list ^^)

1. attach the USB device to Qubes

This is done in L0-dom0, in the Qubes config file.
Only the required options are written here, other options are like any domU. No need for VIFs though, they’re useless in Qubes L1-dom0.

name="qubes41-nested"
type="hvm"
# both required for nested virt
nestedhvm = 1
hap = 1
# USB virtual controller to host the passthrough-ed device ...
usbctrl=[ 'type=devicemodel,version=2,ports=6' ]
# ... and the USB Ethernet device
# Get hostbus and hostaddr from L0 lsusb, and never use leading zeroes
usbdev=[ 'hostbus=1, hostaddr=3, controller=0, port=1' ]

2. allow the USB device through USBguard

By default, Qubes will prevent USB devices to be detected in L1-dom0.
Create a config file “/etc/usbguard/rules.d/99-zusb-ether.conf”

# allow the USB ethernet device - use dmesg or lsusb in Qubes L1 to get that info
allow VID:PID
# example :
# allow 1234:5678

Reboot Qubes, as it does not seem restarting USBguard (via systemctl) is enough in this case.
I’ve created a modprobe file to load the drivers/modules, but it doesn’t seem useful, Qubes does it by default.

3. create a network stack in Qubes L1

Create a bridge in Qubes L1-dom0 (w/o IP), and link it to the USB Ethernet device. This bridge will be used as the sys-net external NIC.

# create an empty bridge
brctl addbr xbr-wan
# set forwarding delay to 0
brctl setfd xbr-wan 0
# bridge the USB and the ... bridge
# note: enp6u4 is the name of the Ethernet USB device (ip a) as detected by Qubes
brctl addif xbr-wan ens6u4
# bring up both ifaces
ip link set enp6u4 up
ip link set xbr-wan up

I know this is the boring way, as it has to be done on each boot, but I’m a fedora noob, didn’t take the time yet to read the correct way (where’s my /etc/network/interface, ffs ? ^^)

4. create a custom config file for sys-net

Now, you need to tell sys-net to use this bridge as an external interface.
I think I found the proper way to do it without messing too much with normal Qubes L2-domU config files.
Please tell me if that’s the recommended way.

Create a file “/etc/qubes/templates/libvirt/xen/by-name/sys-net.xml”.
<!-- --> stanzas are comments which work in the xml file (dunno if # works). They refer to the next line, so you know what’s been done.

<!-- import default template-->
{% extends 'libvirt/xen.xml' %}
<!-- we wanna alter the devices block -->
{% block devices %}
    <!-- import default device block -->
    {{ super() }}
    <!-- manual iface config, we use a bridge -->
    <interface type='bridge'>
        <!-- name of the bridge in Qubes dom0 -->
        <source bridge='xbr-wan' />
        <!-- name of the VIF in Qubes dom0 -->
        <target dev='sys-net-wan'/>
        <!-- public MAC address - this is where you would set external MAC spoofing -->
        <!-- Use whatever hex digits you want, here 00:16:3e is Xen's manufacturer OUI -->
        <mac address='00:16:3e:00:00:01'/>
    </interface>
{% endblock %}

This interface will be detected as eth0 in sys-net.
Now you just have to configure sys-net like usual, ie tell it to use eth0 as the external iface, as you would with any normal NIC.

Conclusion

So, bottom line, it has a few security holes (USB device from L0-dom0 to L2-domU through L1-dom0, a bridge on L1-dom0, only PV L2-domUs, etc), but it works !

3 Likes