If there wasn’t a PV mode, a lot of USB controllers couldn’t/wouldn’t work/be used in Qubes via sys-usb…
Sys-usb in 4.0/4.1 is HVM.
And as far as I’m aware, under 4.0 and 4.1, VM’s that utilize sys-usb’s shared devices are PVH (aka PVHVM), not PV.
I meant on something like this.
Ah, ok. Right.
With R4.0 we saw use of PV mode being deprecated for IOMMU-less systems, but the installer allowed installing without IOMMU, and PV mode was the only way to passthrough PCI (less-safely).
However, R4.1 now requires a functioning IOMMU to install.
Are there that many USB controllers on systems with a functioning IOMMU that are incompatible with the IOMMU?**
** I shouldn’t be surprised if the answer is yes, I suppose.
However, R4.1 now requires a functioning IOMMU to install.
Not quite true, the hard dependency was removed somewhere in the alpha iterations, you can proceed without IOMMU (the installer just show a warning)
However, sys-net / sys-usb will fail to launch as HVM without IOMMU, and if you try to change to PV, it will also fail to launch, because you need to explicitly enable this kernel flag:
So besides the less security of PV, for system without IOMMU at least they can enjoy other Qubes features.
Thanks all for the links and information ! I’ll see what works best for me ^^ For now I don’t really need PCI PT as I’m using X on dom0. I don’t know if nested PCI PT is even possible. I’d only need it for audio, but why is audio handled by a separate domain in Qubes, security ? For now I use
soundhw="hda" in domU config and
pulseaudio/pax11publish to play in dom0.
Anyways, I finally managed to install and run Qubes 4.0.4 in a HVM domU, and I have questions :
- how does the IOMMU work in case of nested virt ? I -think- I have one in dom0 (
iommu: Default domain type: Translated [\n] AMD-Vi: AMD IOMMUv2 functionality not available on this system), but Qubes install reports I don’t have it.
ls /sys/kernel/iommu_groupsis empty on dom0.
- is “
hap = 1” needed ? I don’t know what it does really, but xen wiki/manpages tell to use it with nested virt. I’ve not seen (yet ?) any difference with or without, I even think it’s at 1 by default anyways. Although since I activated it in qubes.cfg, I have many lines like this in xenstored.log: “
data/meminfo_free 2405480 [\n] control/feature-balloon 1”, don’t know if it’s related or not. Autoballooning is disabled in my dom0 (and by default in qubes’), and I use the same values for
maxin both xen cmd lines. Well, I changed it in qubes only AFTER the first boot.
- when qubes boots, I get a few “
(XEN) d16v0: Invalid EFER update: 0x1d01 -> 0x3901 - Unknown bits set” lines, what does that mean ?
- when qubes run, the dom0
xl dmesglog is filling with “
(XEN) d53v1 Unexpected nested vmexit: reason 0x6e” and “
[...] reason 0x87”, to the point that the command never stops ! Where can I read what those error messages mean ? And how can I increase
xl dmesgsize ? I lost all previous pertinent logs …
- should I run qubes domU in HVM or PVH mode ?
- do you know someone who tests nested virt ? ^^
- off-topic, but all my attempts at installing qubes from a NFS-backed ISO failed with “
Payload SHA256 digest: BAD” on various packages, any idea why ?
And please, don’t stop discussing, I learn a lot ; )
I see. Last i thought I read, it was the intention to remove it in R4.1, appears not to be the case though.
Based on my tests I can confirm the warning is the same between 4.0.4 and 4.1.0, it’s just a warning and one can continue the installation, which I find is the correct way to go (more adoption) !
Note that I still can’t install 4.1.0 cause of a luks problem, even if at some point I succeeded but with another (now solved) problem, unfortunately I don’t remember how … So for now I only test 4.0.4.
Now, no one for any of my questions ? ^^
All questions are still valid, but here are some updates (the numbering reflects the order in my previous post) :
- the note about
data/meminfo_free 2405480 [\n] control/feature-balloon 1is irrelevant, I checked the logs and it happens all the time. But the initial point is still valid: is
- the note about
- I still don’t know what the
vmexiterrors mean, but I found a way to log
xl dmesgin text files. In the Xen source code, the
xl dmesgring buffer size is hardcoded to 16k. The solution is to set
/etc/default/xen(that would be sysconfig in Qubes), as per this post, but I have yet to make it work, seems the SIGHUP is not enough.
- I still don’t know what the
vmexit thing seems the most annoying for now. But Qubes domUs seem to work, even if I only have sys-net and sys-fw running.
Ok, “Qubes dom0 as a domU” v4.1 installed and running ! I could even start the sys-net domU, but only in PV mode (HVM = 100% CPU usage and PVH = crash on boot).
I also realized the
vmexit log lines filling my dom0
xl dmesg were caused by using a qubes domU in HVM mode. In PV mode, all good !
Also, for point number 2 about
hap=1, the answer was on the “nested virt” page of the Xen wiki …
For now the requirements I found mandatory to have a nested dom0 working:
# builder="hvm" # deprecated type="hvm" nesthvm=1 hap=1 # using PV drivers notation for disks is ok (maybe because it falls back to hda ? didn't check yet) ... disk = [ 'qubes4.1/disk0.img,,xvda' , ',,xvdb,cdrom' , ] # ... but not for the network, the "vif" is not detected by qubes dom0, but the "e1000" is vif = [ 'bridge=xbr-pftests, mac=00:16:3e:48:40:40, vifname=qubes41,model=e1000' , 'bridge=xbr-pftests, mac=00:16:3e:48:40:41, vifname=qubes41-pv,type=vif' , ]
So I’m left with:
- (old 1.) trying to see if I have a IOMMU and if yes get it to work (BIOS and/or microcode update ?)
- (old 3.) Invalid EFER update, I have as many lines as I have vCPU given to qubes
- this bug (qubes not bootable cause 100% CPU), but not sure if it’s related to a nested virt prob or another, no log avail
- when from Qubes dom0 I do
xl console DOMAIN(as I can’t terminal in yet …), the console is really awful (lots of control characters printed), why is that ?
I always tend to ask myself
why, instead of
when instead of
if, so while this sounds interesting, why we would want to run Qubes in domU under Xen as dom0?
Ahah, yeah nice way to think, I agree ^^ Even if IT, for me, is not always “I need this”, but “I want to test this” ! When you’re growing up, only the toys change, not the mentality ^^
Anyways, there are many reasons ! Some are related to my specific use case, some may be more general.
Specific to me :
- First and foremost, because I already have a dom0 host, and I want to try Qubes ! Also I don’t have another capable and powerful enough machine to run Qubes.
- As explained in this post (read only the 2 first §), my dom0 hosts domUs/services used by my local network.
- I can “port” Qubes functionalities into my system to render it more usable and secure, as well as learning new management tricks about Xen and systems management.
- Why not ? ^^ I like tinkering and discovering new things !
More genericly :
- people often try new OS in virtual machines, so I thought people could benefit from my tests. But ok, I admit Xen is not the typical hypervisor user choice !
- related, testing Qubes from Qubes may be useful to Qubes developpers and enthousiasts alike, to test new kernels, functionnalities, etc, without breaking the main install (even though not everything could be validated from a Qubes domU).
- I’ve read Qubes is thinking about the cloud (but if I understood correctly, not for dom0 rather the domUs), and some providers use Xen as their hypervisor, so it would allow to test/run a full Qubes dom0 there. Some users could use Tails/Whonix on local hardware to connect to a cloud-based Qubes dom0.
- testing nested virt provides the Xen project more information/debug about this experimental feature.
And this one is a mix of both reasons:
- Qubes is a “user oriented” system, so it makes sense to run it virtualized on a more “infrastructure oriented” system (like mine), even if I agree that it poses security risks that Qubes may not be able to handle (but should try to ?). Even if Qubes’ main goal looks to be the ultimate most secure OS (why not openBSD-based btw ?), it is already used by people who don’t care about EVERY security aspects of it. And to me having more users is a good thing for a distro, so the more use cases Qubes cover, the merrier ! With all due warnings of course.
There are maybe things I dont think of right now, but I guess it’s enough good reasons, no ?
Sorry long post, but you asked … ^^
PS to moderators: could our 2 last posts be split into another discussion in “General discussion” please ? It would be more relevant than here (User support 4.1) and I’d like to keep this post related to the technical aspects of it. Or move the whole conversation, and I’ll create new posts about it ? As you want, TIA !
Thanks for your response, I find it very useful for all that might read it, and I must say that somehow I felt that “temptation to make it run as Domu” was one of the main reasons. While until qubes can be run only in PV mode I don’t see it’s full potential can be used, still more than interested to follow your findings. Thanks for being first to write, if not to test, this.
Let’s flag the topic, that should be the fastest way to summon mods.
Thanks for the nice comments, I hope it attracts more attention as I think the use cases are interesting, and not only for my little person ! ^^
Thanks, but first, oh no ^^ Look those posts, from 2015/2016 !
- How to nested virtualization with Xen?
- [Advanced] Enabling nested virtualization in Qubes/HVM
- nested Qubes (Qubes within Qubes) can work - proof of concept ^^
This is totally uncharted territory for me, dunno what you mean ^^
… to call the mods to split the topic…
Ah, you mean I flag my own post with a message to ask for splitting, so I can create a new topic which could be titled “Why running Qubes as a domU from another dom0 can be useful” ?
Hahah, I flagged whole topic asking them to split the topic by your suggestion
Ah ah, ok, thanks then ^^ I don’t know how that works nowadays, but let’s see, you learn something new everyday (was the phpbb2 dev/admin for my gaming clan/friends) !
Sorry, I’ve been trying to split this, but I lack context and the discussion goes beyond my technical knowledge. So I’m afraid I can’t help much.
So it would be better if you create another thread in general discussion summarizing the discussion the discussion stemmed from here (for example via a linked topic).
Also, when requesting a topic split, a way to do it is to flag the topic as @enmus suggested, but also saying which posts you want to split and what’s the new topic name. Without that key context and especially on highly technical topics it can be hard for mods to know.
Thanks. I’ll keep that in mind on occasions splitting would be my idea.