Sys-usb in 4.0/4.1 is HVM.
And as far as I’m aware, under 4.0 and 4.1, VM’s that utilize sys-usb’s shared devices are PVH (aka PVHVM), not PV.
Brendan
Sys-usb in 4.0/4.1 is HVM.
And as far as I’m aware, under 4.0 and 4.1, VM’s that utilize sys-usb’s shared devices are PVH (aka PVHVM), not PV.
Brendan
Ah, ok. Right.
With R4.0 we saw use of PV mode being deprecated for IOMMU-less systems, but the installer allowed installing without IOMMU, and PV mode was the only way to passthrough PCI (less-safely).
However, R4.1 now requires a functioning IOMMU to install.
Are there that many USB controllers on systems with a functioning IOMMU that are incompatible with the IOMMU?**
B
** I shouldn’t be surprised if the answer is yes, I suppose.
However, R4.1 now requires a functioning IOMMU to install.
Not quite true, the hard dependency was removed somewhere in the alpha iterations, you can proceed without IOMMU (the installer just show a warning)
However, sys-net / sys-usb will fail to launch as HVM without IOMMU, and if you try to change to PV, it will also fail to launch, because you need to explicitly enable this kernel flag: qubes.enable_insecure_pv_passthrough
(commit)
So besides the less security of PV, for system without IOMMU at least they can enjoy other Qubes features.
Thanks all for the links and information ! I’ll see what works best for me ^^ For now I don’t really need PCI PT as I’m using X on dom0. I don’t know if nested PCI PT is even possible. I’d only need it for audio, but why is audio handled by a separate domain in Qubes, security ? For now I use soundhw="hda"
in domU config and pulseaudio/pax11publish
to play in dom0.
Anyways, I finally managed to install and run Qubes 4.0.4 in a HVM domU, and I have questions :
iommu: Default domain type: Translated [\n] AMD-Vi: AMD IOMMUv2 functionality not available on this system
), but Qubes install reports I don’t have it. ls /sys/kernel/iommu_groups
is empty on dom0.hap = 1
” needed ? I don’t know what it does really, but xen wiki/manpages tell to use it with nested virt. I’ve not seen (yet ?) any difference with or without, I even think it’s at 1 by default anyways. Although since I activated it in qubes.cfg, I have many lines like this in xenstored.log: “data/meminfo_free 2405480 [\n] control/feature-balloon 1
”, don’t know if it’s related or not. Autoballooning is disabled in my dom0 (and by default in qubes’), and I use the same values for dom0_mem
and max
in both xen cmd lines. Well, I changed it in qubes only AFTER the first boot.(XEN) d16v0: Invalid EFER update: 0x1d01 -> 0x3901 - Unknown bits set
” lines, what does that mean ?xl dmesg
log is filling with “(XEN) d53v1 Unexpected nested vmexit: reason 0x6e
” and “[...] reason 0x87
”, to the point that the command never stops ! Where can I read what those error messages mean ? And how can I increase xl dmesg
size ? I lost all previous pertinent logs …Payload SHA256 digest: BAD
” on various packages, any idea why ?And please, don’t stop discussing, I learn a lot ; )
I see. Last i thought I read, it was the intention to remove it in R4.1, appears not to be the case though.
B
Based on my tests I can confirm the warning is the same between 4.0.4 and 4.1.0, it’s just a warning and one can continue the installation, which I find is the correct way to go (more adoption) !
Note that I still can’t install 4.1.0 cause of a luks problem, even if at some point I succeeded but with another (now solved) problem, unfortunately I don’t remember how … So for now I only test 4.0.4.
Now, no one for any of my questions ? ^^
All questions are still valid, but here are some updates (the numbering reflects the order in my previous post) :
data/meminfo_free 2405480 [\n] control/feature-balloon 1
is irrelevant, I checked the logs and it happens all the time. But the initial point is still valid: is hap
neededvmexit
errors mean, but I found a way to log xl dmesg
in text files. In the Xen source code, the xl dmesg
ring buffer size is hardcoded to 16k. The solution is to set XENCONSOLED_TRACE
in /etc/default/xen
(that would be sysconfig in Qubes), as per this post, but I have yet to make it work, seems the SIGHUP is not enough.The vmexit
thing seems the most annoying for now. But Qubes domUs seem to work, even if I only have sys-net and sys-fw running.
Ok, “Qubes dom0 as a domU” v4.1 installed and running ! I could even start the sys-net domU, but only in PV mode (HVM = 100% CPU usage and PVH = crash on boot).
I also realized the vmexit
log lines filling my dom0 xl dmesg
were caused by using a qubes domU in HVM mode. In PV mode, all good !
Also, for point number 2 about hap=1
, the answer was on the “nested virt” page of the Xen wiki …
For now the requirements I found mandatory to have a nested dom0 working:
# builder="hvm" # deprecated
type="hvm"
nesthvm=1
hap=1
# using PV drivers notation for disks is ok (maybe because it falls back to hda ? didn't check yet) ...
disk = [ 'qubes4.1/disk0.img,,xvda' ,
',,xvdb,cdrom' ,
]
# ... but not for the network, the "vif" is not detected by qubes dom0, but the "e1000" is
vif = [ 'bridge=xbr-pftests, mac=00:16:3e:48:40:40, vifname=qubes41,model=e1000' ,
'bridge=xbr-pftests, mac=00:16:3e:48:40:41, vifname=qubes41-pv,type=vif' ,
]
So I’m left with:
xl console DOMAIN
(as I can’t terminal in yet …), the console is really awful (lots of control characters printed), why is that ?I always tend to ask myself why
, instead of how
, and when
instead of if
, so while this sounds interesting, why we would want to run Qubes in domU under Xen as dom0?
Ahah, yeah nice way to think, I agree ^^ Even if IT, for me, is not always “I need this”, but “I want to test this” ! When you’re growing up, only the toys change, not the mentality ^^
Anyways, there are many reasons ! Some are related to my specific use case, some may be more general.
Specific to me :
More genericly :
And this one is a mix of both reasons:
There are maybe things I dont think of right now, but I guess it’s enough good reasons, no ?
Sorry long post, but you asked … ^^
PS to moderators: could our 2 last posts be split into another discussion in “General discussion” please ? It would be more relevant than here (User support 4.1) and I’d like to keep this post related to the technical aspects of it. Or move the whole conversation, and I’ll create new posts about it ? As you want, TIA !
Thanks for your response, I find it very useful for all that might read it, and I must say that somehow I felt that “temptation to make it run as Domu” was one of the main reasons. While until qubes can be run only in PV mode I don’t see it’s full potential can be used, still more than interested to follow your findings. Thanks for being first to write, if not to test, this.
Let’s flag the topic, that should be the fastest way to summon mods.
Thanks for the nice comments, I hope it attracts more attention as I think the use cases are interesting, and not only for my little person ! ^^
Thanks, but first, oh no ^^ Look those posts, from 2015/2016 !
This is totally uncharted territory for me, dunno what you mean ^^
… to call the mods to split the topic…
Ah, you mean I flag my own post with a message to ask for splitting, so I can create a new topic which could be titled “Why running Qubes as a domU from another dom0 can be useful” ?
Hahah, I flagged whole topic asking them to split the topic by your suggestion
Ah ah, ok, thanks then ^^ I don’t know how that works nowadays, but let’s see, you learn something new everyday (was the phpbb2 dev/admin for my gaming clan/friends) !
Sorry, I’ve been trying to split this, but I lack context and the discussion goes beyond my technical knowledge. So I’m afraid I can’t help much.
So it would be better if you create another thread in general discussion summarizing the discussion the discussion stemmed from here (for example via a linked topic).
Also, when requesting a topic split, a way to do it is to flag the topic as @enmus suggested, but also saying which posts you want to split and what’s the new topic name. Without that key context and especially on highly technical topics it can be hard for mods to know.
Thanks. I’ll keep that in mind on occasions splitting would be my idea.
So, I finally got a working nested Qubes ! I mean, connected to the network.
The trick is to use a USB Ethernet adapter, as USB passthrough works, on the contrary to PCI.
WARNING : this method is only for testing purposes !!!
Except for non-PV L2-domUs, Qubes run fine nested, and works as expected.
The hard part is to get a working external network connection.
I solved it by using an USB Ethernet device. It is connected in L0-dom0, and USB passthrough-ed to Qubes L1.
There are 4 main steps :
(hack, sorry, didn't find an elegant way to reset a numbered list ^^)
This is done in L0-dom0, in the Qubes config file.
Only the required options are written here, other options are like any domU. No need for VIFs though, they’re useless in Qubes L1-dom0.
name="qubes41-nested"
type="hvm"
# both required for nested virt
nestedhvm = 1
hap = 1
# USB virtual controller to host the passthrough-ed device ...
usbctrl=[ 'type=devicemodel,version=2,ports=6' ]
# ... and the USB Ethernet device
# Get hostbus and hostaddr from L0 lsusb, and never use leading zeroes
usbdev=[ 'hostbus=1, hostaddr=3, controller=0, port=1' ]
By default, Qubes will prevent USB devices to be detected in L1-dom0.
Create a config file “/etc/usbguard/rules.d/99-zusb-ether.conf”
# allow the USB ethernet device - use dmesg or lsusb in Qubes L1 to get that info
allow VID:PID
# example :
# allow 1234:5678
Reboot Qubes, as it does not seem restarting USBguard (via systemctl
) is enough in this case.
I’ve created a modprobe file to load the drivers/modules, but it doesn’t seem useful, Qubes does it by default.
Create a bridge in Qubes L1-dom0 (w/o IP), and link it to the USB Ethernet device. This bridge will be used as the sys-net external NIC.
# create an empty bridge
brctl addbr xbr-wan
# set forwarding delay to 0
brctl setfd xbr-wan 0
# bridge the USB and the ... bridge
# note: enp6u4 is the name of the Ethernet USB device (ip a) as detected by Qubes
brctl addif xbr-wan ens6u4
# bring up both ifaces
ip link set enp6u4 up
ip link set xbr-wan up
I know this is the boring way, as it has to be done on each boot, but I’m a fedora noob, didn’t take the time yet to read the correct way (where’s my /etc/network/interface, ffs ? ^^)
Now, you need to tell sys-net to use this bridge as an external interface.
I think I found the proper way to do it without messing too much with normal Qubes L2-domU config files.
Please tell me if that’s the recommended way.
Create a file “/etc/qubes/templates/libvirt/xen/by-name/sys-net.xml”.
<!-- -->
stanzas are comments which work in the xml file (dunno if #
works). They refer to the next line, so you know what’s been done.
<!-- import default template-->
{% extends 'libvirt/xen.xml' %}
<!-- we wanna alter the devices block -->
{% block devices %}
<!-- import default device block -->
{{ super() }}
<!-- manual iface config, we use a bridge -->
<interface type='bridge'>
<!-- name of the bridge in Qubes dom0 -->
<source bridge='xbr-wan' />
<!-- name of the VIF in Qubes dom0 -->
<target dev='sys-net-wan'/>
<!-- public MAC address - this is where you would set external MAC spoofing -->
<!-- Use whatever hex digits you want, here 00:16:3e is Xen's manufacturer OUI -->
<mac address='00:16:3e:00:00:01'/>
</interface>
{% endblock %}
This interface will be detected as eth0 in sys-net.
Now you just have to configure sys-net like usual, ie tell it to use eth0 as the external iface, as you would with any normal NIC.
So, bottom line, it has a few security holes (USB device from L0-dom0 to L2-domU through L1-dom0, a bridge on L1-dom0, only PV L2-domUs, etc), but it works !