Installing Qubes 4.1 in a Xen HVM domU (nestedhvm=1)

User Support General
25

So, I finally got a working nested Qubes ! I mean, connected to the network.
The trick is to use a USB Ethernet adapter, as USB passthrough works, on the contrary to PCI.
WARNING : this method is only for testing purposes !!!

Terminology

  • L0-dom0 : this is the dom0 installed on bare metal, a vanilla Xen on Debian for me, no libvirt.
  • L1-dom0 : this is the Qubes install, which is a HVM domU running on L0-dom0. I’ll refer to it as Qubes or Qubes dom0 or Qubes L1.
  • L2-domU : all the domUs started by Qubes (L1-)dom0. So sys-net, sys-firewall, etc

Problems/limits

  • only PV L2-domUs are supported. PVH and HVM crash Qubes L1-dom0 (vmexit errors reported in L0-dom0)
  • PCI passthrough-ing emulated hardware (like the QEMU e1000) from Qubes L1 to L2-domUs won’t work.

How-to steps

Except for non-PV L2-domUs, Qubes run fine nested, and works as expected.
The hard part is to get a working external network connection.
I solved it by using an USB Ethernet device. It is connected in L0-dom0, and USB passthrough-ed to Qubes L1.

There are 4 main steps :

  1. attach the USB device from L0-dom0 to Qubes L1
  2. allow the USB device through USBguard
  3. create a network stack in Qubes L1
  4. create a custom config file for L2 sys-net
(hack, sorry, didn't find an elegant way to reset a numbered list ^^)

1. attach the USB device to Qubes

This is done in L0-dom0, in the Qubes config file.
Only the required options are written here, other options are like any domU. No need for VIFs though, they’re useless in Qubes L1-dom0.

name="qubes41-nested"
type="hvm"
# both required for nested virt
nestedhvm = 1
hap = 1
# USB virtual controller to host the passthrough-ed device ...
usbctrl=[ 'type=devicemodel,version=2,ports=6' ]
# ... and the USB Ethernet device
# Get hostbus and hostaddr from L0 lsusb, and never use leading zeroes
usbdev=[ 'hostbus=1, hostaddr=3, controller=0, port=1' ]

2. allow the USB device through USBguard

By default, Qubes will prevent USB devices to be detected in L1-dom0.
Create a config file “/etc/usbguard/rules.d/99-zusb-ether.conf”

# allow the USB ethernet device - use dmesg or lsusb in Qubes L1 to get that info
allow VID:PID
# example :
# allow 1234:5678

Reboot Qubes, as it does not seem restarting USBguard (via systemctl) is enough in this case.
I’ve created a modprobe file to load the drivers/modules, but it doesn’t seem useful, Qubes does it by default.

3. create a network stack in Qubes L1

Create a bridge in Qubes L1-dom0 (w/o IP), and link it to the USB Ethernet device. This bridge will be used as the sys-net external NIC.

# create an empty bridge
brctl addbr xbr-wan
# set forwarding delay to 0
brctl setfd xbr-wan 0
# bridge the USB and the ... bridge
# note: enp6u4 is the name of the Ethernet USB device (ip a) as detected by Qubes
brctl addif xbr-wan ens6u4
# bring up both ifaces
ip link set enp6u4 up
ip link set xbr-wan up

I know this is the boring way, as it has to be done on each boot, but I’m a fedora noob, didn’t take the time yet to read the correct way (where’s my /etc/network/interface, ffs ? ^^)

4. create a custom config file for sys-net

Now, you need to tell sys-net to use this bridge as an external interface.
I think I found the proper way to do it without messing too much with normal Qubes L2-domU config files.
Please tell me if that’s the recommended way.

Create a file “/etc/qubes/templates/libvirt/xen/by-name/sys-net.xml”.
<!-- --> stanzas are comments which work in the xml file (dunno if # works). They refer to the next line, so you know what’s been done.

<!-- import default template-->
{% extends 'libvirt/xen.xml' %}
<!-- we wanna alter the devices block -->
{% block devices %}
    <!-- import default device block -->
    {{ super() }}
    <!-- manual iface config, we use a bridge -->
    <interface type='bridge'>
        <!-- name of the bridge in Qubes dom0 -->
        <source bridge='xbr-wan' />
        <!-- name of the VIF in Qubes dom0 -->
        <target dev='sys-net-wan'/>
        <!-- public MAC address - this is where you would set external MAC spoofing -->
        <!-- Use whatever hex digits you want, here 00:16:3e is Xen's manufacturer OUI -->
        <mac address='00:16:3e:00:00:01'/>
    </interface>
{% endblock %}

This interface will be detected as eth0 in sys-net.
Now you just have to configure sys-net like usual, ie tell it to use eth0 as the external iface, as you would with any normal NIC.

Conclusion

So, bottom line, it has a few security holes (USB device from L0-dom0 to L2-domU through L1-dom0, a bridge on L1-dom0, only PV L2-domUs, etc), but it works !

3 Likes