Installing Qubes 4.1 in a Xen HVM domU (nestedhvm=1)

User Support General
21

Hahah, I flagged whole topic asking them to split the topic by your suggestion

22

Ah ah, ok, thanks then ^^ I don’t know how that works nowadays, but let’s see, you learn something new everyday (was the phpbb2 dev/admin for my gaming clan/friends) !

23

Sorry, I’ve been trying to split this, but I lack context and the discussion goes beyond my technical knowledge. So I’m afraid I can’t help much.

So it would be better if you create another thread in general discussion summarizing the discussion the discussion stemmed from here (for example via a linked topic).

Also, when requesting a topic split, a way to do it is to flag the topic as @enmus suggested, but also saying which posts you want to split and what’s the new topic name. Without that key context and especially on highly technical topics it can be hard for mods to know.

1 Like
24

Thanks. I’ll keep that in mind on occasions splitting would be my idea.

1 Like
25

So, I finally got a working nested Qubes ! I mean, connected to the network.
The trick is to use a USB Ethernet adapter, as USB passthrough works, on the contrary to PCI.
WARNING : this method is only for testing purposes !!!

Terminology

  • L0-dom0 : this is the dom0 installed on bare metal, a vanilla Xen on Debian for me, no libvirt.
  • L1-dom0 : this is the Qubes install, which is a HVM domU running on L0-dom0. I’ll refer to it as Qubes or Qubes dom0 or Qubes L1.
  • L2-domU : all the domUs started by Qubes (L1-)dom0. So sys-net, sys-firewall, etc

Problems/limits

  • only PV L2-domUs are supported. PVH and HVM crash Qubes L1-dom0 (vmexit errors reported in L0-dom0)
  • PCI passthrough-ing emulated hardware (like the QEMU e1000) from Qubes L1 to L2-domUs won’t work.

How-to steps

Except for non-PV L2-domUs, Qubes run fine nested, and works as expected.
The hard part is to get a working external network connection.
I solved it by using an USB Ethernet device. It is connected in L0-dom0, and USB passthrough-ed to Qubes L1.

There are 4 main steps :

  1. attach the USB device from L0-dom0 to Qubes L1
  2. allow the USB device through USBguard
  3. create a network stack in Qubes L1
  4. create a custom config file for L2 sys-net
(hack, sorry, didn't find an elegant way to reset a numbered list ^^)

1. attach the USB device to Qubes

This is done in L0-dom0, in the Qubes config file.
Only the required options are written here, other options are like any domU. No need for VIFs though, they’re useless in Qubes L1-dom0.

name="qubes41-nested"
type="hvm"
# both required for nested virt
nestedhvm = 1
hap = 1
# USB virtual controller to host the passthrough-ed device ...
usbctrl=[ 'type=devicemodel,version=2,ports=6' ]
# ... and the USB Ethernet device
# Get hostbus and hostaddr from L0 lsusb, and never use leading zeroes
usbdev=[ 'hostbus=1, hostaddr=3, controller=0, port=1' ]

2. allow the USB device through USBguard

By default, Qubes will prevent USB devices to be detected in L1-dom0.
Create a config file “/etc/usbguard/rules.d/99-zusb-ether.conf”

# allow the USB ethernet device - use dmesg or lsusb in Qubes L1 to get that info
allow VID:PID
# example :
# allow 1234:5678

Reboot Qubes, as it does not seem restarting USBguard (via systemctl) is enough in this case.
I’ve created a modprobe file to load the drivers/modules, but it doesn’t seem useful, Qubes does it by default.

3. create a network stack in Qubes L1

Create a bridge in Qubes L1-dom0 (w/o IP), and link it to the USB Ethernet device. This bridge will be used as the sys-net external NIC.

# create an empty bridge
brctl addbr xbr-wan
# set forwarding delay to 0
brctl setfd xbr-wan 0
# bridge the USB and the ... bridge
# note: enp6u4 is the name of the Ethernet USB device (ip a) as detected by Qubes
brctl addif xbr-wan ens6u4
# bring up both ifaces
ip link set enp6u4 up
ip link set xbr-wan up

I know this is the boring way, as it has to be done on each boot, but I’m a fedora noob, didn’t take the time yet to read the correct way (where’s my /etc/network/interface, ffs ? ^^)

4. create a custom config file for sys-net

Now, you need to tell sys-net to use this bridge as an external interface.
I think I found the proper way to do it without messing too much with normal Qubes L2-domU config files.
Please tell me if that’s the recommended way.

Create a file “/etc/qubes/templates/libvirt/xen/by-name/sys-net.xml”.
<!-- --> stanzas are comments which work in the xml file (dunno if # works). They refer to the next line, so you know what’s been done.

<!-- import default template-->
{% extends 'libvirt/xen.xml' %}
<!-- we wanna alter the devices block -->
{% block devices %}
    <!-- import default device block -->
    {{ super() }}
    <!-- manual iface config, we use a bridge -->
    <interface type='bridge'>
        <!-- name of the bridge in Qubes dom0 -->
        <source bridge='xbr-wan' />
        <!-- name of the VIF in Qubes dom0 -->
        <target dev='sys-net-wan'/>
        <!-- public MAC address - this is where you would set external MAC spoofing -->
        <!-- Use whatever hex digits you want, here 00:16:3e is Xen's manufacturer OUI -->
        <mac address='00:16:3e:00:00:01'/>
    </interface>
{% endblock %}

This interface will be detected as eth0 in sys-net.
Now you just have to configure sys-net like usual, ie tell it to use eth0 as the external iface, as you would with any normal NIC.

Conclusion

So, bottom line, it has a few security holes (USB device from L0-dom0 to L2-domU through L1-dom0, a bridge on L1-dom0, only PV L2-domUs, etc), but it works !

3 Likes