Hello, not currently a Qubes user yet but wanting to, I discovered it after having done a somewhat similar system (but WAY less brilliant and secure !), also based on Xen.
I wanted a “Network-in-a-box” setup to avoid using multiple physical machines (ecology ^^), so I went the route of: Debian dom0 with X(fce), pfsense as a global router/fw, FreeNAS as a global datastore/NAS, a Debian VM for netservices (DNS, NTP, etc), and a few user domUs for apps, one of which is a gaming U with GPU passthrough. You can check my setup here.
dom0 has no direct internet access (except when updating), and only uses “ssh -X” to connect to apps on other Us. The WAN netcard is passthrough’ed to the pfsense machine.
I’ve read a lot of articles in the documentation, but to understand “the Qubes way” fully I have several questions. Don’t hesitate to point me to the documentation i may have missed/read too fast !
My ultimate setup may look like this :
- using a dom0 for hosting infrastructure VMs for my whole private network: pfsense as router/edge fw, freeNAS as data storage, and an openBSD for netservices
- using Qubes as a domU for all user-related things
- i don’t have IOMMU groups (Ryzen 1700X 8c/16t), what are the security implications ?
- is there a way to PT safely ? Is the driver domain using PT ? (note: I don’t fully understand yet how Xen stub/driver domains work)
- I’ve read in the docs that nested virtualization is not supported, but Xen manpages tell that a dom0 can run inside a domU. Are the docs only talking about non-Xen nested virtualization ?
- is XSM/FLASK supported ? I just discovered it in Xen manpages, it seems a nice security feature
- would it be possible to use pfSense (or alike) as the firewallVM ?
- i’ve read in the docs " In order to eliminate layer 2 attacks originating from a compromised VM, routed networking is used instead of the default bridging of
vifdevices and NAT is applied at each network hop". What I don’t get is that if dom0 has no network stack or access to it, how is it a problem ? To understand why I ask this I need to explain my current setup: for each machine that needs network access, I create a bridge (w/o IP, bridge_stp off and bridge_fd 0) on dom0, and on each bridge I put a pfSense vif and the machine vif. The machine then goes through pfsense to access all other machines (including dom0). It seems layer2 discovery is not possible anymore on dom0 (see for example this link). How do my way compare to Qubes network setup ?
- has root-on-ZFS been tested in Qubes dom0 ?
- would a freeNAS domU be able to handle storage for VM templates and/or data ?
- how is seamless mode handled for a Win VM, how do QWT works and how can I adapt it to my vanilla Xen ? I want to learn how it’s done in Qubes ! My win7 domU has a PT GPU and I’m fed up with manual screen switching, and VNC is a bit a pain to use with multiple screens.
PS: sorry for the long post, and should I split this post into several ones for easier discussion ?
Thanks and have a nice day !