Hello, I have qubes connected to a trunk port in my switch, and would like to configure multiple VLANs support on sys-net, where each vlan interface is assigned to a custom sys-firewall to restrict AppVMs network access to their sys-firewall vlan. Here’s a diagram of what I’m trying to accomplish:
I’ve searched around but so far didn’t find any guide to work on this, I see many related scripts in /etc/xen/scripts but would like any insight in how to use these both in sys-net & sys-firewallX to accomplish this (Untagged traffic would be removed by bringing the main interface off).
Just to clarify, was a trunk link established between the end-device (sys-net) and the switch? If yes, what kind of NIC do you have?
Yes, ethernet adapter
How many network controllers do you have?
1
Why would you want sys-net handling trunking?
To have multiple sys-firewall, each one attached to one VLAN interface so I can isolate AppVMs connected to these sys-firewall to their specified VLAN.
What’s the current state of your setup? Do you have net access?
Have you done any kind of dot1q encapsulation in your sys-net or is it supposed to be natively supported by your NIC?
Generally speaking, trunking on an end-device is not a great idea, so perhaps, depending on your situation, it may be worth exploring purchasing a second NIC and attaching it to a different sys-net vm. That way you’d be effectively separating VLAN traffic into two different vms (sys-net-10 and sys-net-20).
What’s the current state of your setup? Do you have net access?
Yes, network works fine using untagged traffic for both sys-* and appvms
Have you done any kind of dot1q encapsulation in your sys-net or is it supposed to be natively supported by your NIC?
I can configure VLANs using NetworkManage and they work correctly on sys-net
Generally speaking, trunking on an end-device is not a great idea, so perhaps, depending on your situation, it may be worth exploring purchasing a second NIC and attaching it to a different sys-net vm. That way you’d be effectively separating VLAN traffic into two different vms (sys-net-10 and sys-net-20).
Yes, although in this model sys-net wouldn’t be just an “end-device” as it would provide vlans similar to how the switch is doing it (sys-firewall* would deal only with untagged traffic). For a 2nd nic, I already tested with an external adapter and it worked, however I have way more than 2 VLANs to configure in sys-net (diagram was just a simple example), so can’t go using that route.
Just covering the basics here, sorry if it’s obvious: he indicates to place scripts in /rw/config/rc.local and /rw/config/qubes-firewall-user-script. If your sys-net is a DispVM you’ll need to edit the dvm-template which sys-net is based on, otherwise they won’t be there after a reboot.
Seems I was using sys-net vif ip address instead of sys-firewall eth0 ip address, but I also see that I need to “disconnect” from untagged connection. After doing these 2 changes things work correctly on sys-firewall and its connected appvms.
Will play a bit more and provide some documentation for others in need of a similar setup.
Looking for the promised documentation…
Can you post your iptables?
I have a hybrid port (not trunk or access) with only two vlans: 2 and 3, both tagged.
Both vlans come up. Called “02-mgmt” and “03-general”.
I have created a firewall-02, for the “02-mgmt” vlan.
On /sys-net/ I am trying to create rules. All seem to just turn into 0.0.0/0 for source and destination, no matter what I do.
Firgured out the issue. iptables -L for some reason always confuses. iptables -vL confirms that the settings were being made. I now have one AppVM with a custom firewall VM which is on the desired vlan, everything else on the “general” vlan.
bash-5.2# iptables -vL FORWARD
Chain FORWARD (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
18182 1654K ACCEPT all -- any 02-mgmt 10.138.10.236 anywhere
189 15206 DROP all -- any any 10.138.10.236 anywhere
428 22788 DROP all -- any 02-mgmt anywhere anywhere
0 0 DROP all -- any any anywhere anywhere state INVALID
18265 2582K ACCEPT all -- any any anywhere anywhere ctstate RELATED,ESTABLISHED
77 6158 QBS-FORWARD all -- any any anywhere anywhere
0 0 DROP all -- vif+ vif+ anywhere anywhere
77 6158 ACCEPT all -- vif+ any anywhere anywhere
0 0 DROP all -- any any anywhere anywhere
bash-5.2#
This is the vlan 02 interface:
bash-5.2# ip -4 addr show 02-mgmt
3: 02-mgmt@ens6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
inet 192.168.2.12/24 brd 192.168.2.255 scope global noprefixroute 02-mgmt
valid_lft forever preferred_lft forever
The 10.138.10.236 is the address assigned to the sys-firewall-02-mgmt VM.
Hello @QubesAnon, I actually dropped this and ended up doing some refactoring on VLANs and working with 2 physical nics instead, but glad you figured it out based on unman post.
Hi @equbes. I ended up with something different once I got things fully working…
There were issues regarding default routes being added to both VLANs: had to address that, so I ended up with some nmcli edits and some nm-gui additions and a somewhat simplified set of rules. Putting ip route commands in the /rw/config/rc.local did not work – something flushes all the routes and just added back the dual default routes, so NetworkMunger was where this was addressed.
I also needed an explicit route to the 192.168.1.0/24 vlan1. The vlan2 firewall-attached VMs are supposed to perform management functions on the vlan1-resident infrastructure. I have one VM which performs the management stuff, and the rest of the VMs are supposed to use vlan3. My firewall enforces those access restrictions. The Qubes issues arise because I wish to have VM attach to vlan2 for management and the rest vlan3 for “normal” access. The management VM also can access all the stuff on vlan3.
bash-5.2# iptables -vL LOGGING
Chain LOGGING (2 references)
pkts bytes target prot opt in out source destination
206 37507 LOG all -- any any anywhere anywhere LOG level warning prefix "ipt denied: "
206 37507 DROP all -- any any anywhere anywhere
bash-5.2# iptables -vL FORWARD
Chain FORWARD (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
12711 1689K ACCEPT all -- any 02-mgmt 10.138.10.236 anywhere
75796 7876K ACCEPT all -- any 03-general anywhere anywhere
0 0 LOGGING all -- any any 10.138.10.236 anywhere
206 37507 LOGGING all -- any 02-mgmt anywhere anywhere
0 0 DROP all -- any any anywhere anywhere state INVALID
109K 90M ACCEPT all -- any any anywhere anywhere ctstate RELATED,ESTABLISHED
0 0 QBS-FORWARD all -- any any anywhere anywhere
0 0 DROP all -- vif+ vif+ anywhere anywhere
0 0 ACCEPT all -- vif+ any anywhere anywhere
0 0 DROP all -- any any anywhere anywhere
bash-5.2# ip -4 route
default via 192.168.3.1 dev 03-general proto dhcp src 192.168.3.12 metric 401
10.138.10.236 dev vif10.0 scope link metric 32742
10.138.31.72 dev vif9.0 scope link metric 32743
192.168.1.0/24 via 192.168.2.1 dev 02-mgmt proto static metric 400
192.168.2.0/24 dev 02-mgmt proto kernel scope link src 192.168.2.12 metric 400
192.168.2.1 dev 02-mgmt proto static scope link metric 400
192.168.3.0/24 dev 03-general proto kernel scope link src 192.168.3.12 metric 401
bash-5.2# ip -4 addr
3: 03-general@ens6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
inet 192.168.3.12/24 brd 192.168.3.255 scope global dynamic noprefixroute 03-general
valid_lft 79554sec preferred_lft 79554sec
4: 02-mgmt@ens6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
inet 192.168.2.12/24 brd 192.168.2.255 scope global dynamic noprefixroute 02-mgmt
valid_lft 79554sec preferred_lft 79554sec
5: vif9.0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 32
inet 10.137.0.6/32 scope global vif9.0
valid_lft forever preferred_lft forever
6: vif10.0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 32
inet 10.137.0.6/32 scope global vif10.0
valid_lft forever preferred_lft forever
bash-5.2#