Hello, I have qubes connected to a trunk port in my switch, and would like to configure multiple VLANs support on sys-net, where each vlan interface is assigned to a custom sys-firewall to restrict AppVMs network access to their sys-firewall vlan. Here’s a diagram of what I’m trying to accomplish:
I’ve searched around but so far didn’t find any guide to work on this, I see many related scripts in /etc/xen/scripts but would like any insight in how to use these both in sys-net & sys-firewallX to accomplish this (Untagged traffic would be removed by bringing the main interface off).
Just to clarify, was a trunk link established between the end-device (sys-net) and the switch? If yes, what kind of NIC do you have?
Yes, ethernet adapter
How many network controllers do you have?
1
Why would you want sys-net handling trunking?
To have multiple sys-firewall, each one attached to one VLAN interface so I can isolate AppVMs connected to these sys-firewall to their specified VLAN.
What’s the current state of your setup? Do you have net access?
Have you done any kind of dot1q encapsulation in your sys-net or is it supposed to be natively supported by your NIC?
Generally speaking, trunking on an end-device is not a great idea, so perhaps, depending on your situation, it may be worth exploring purchasing a second NIC and attaching it to a different sys-net vm. That way you’d be effectively separating VLAN traffic into two different vms (sys-net-10 and sys-net-20).
What’s the current state of your setup? Do you have net access?
Yes, network works fine using untagged traffic for both sys-* and appvms
Have you done any kind of dot1q encapsulation in your sys-net or is it supposed to be natively supported by your NIC?
I can configure VLANs using NetworkManage and they work correctly on sys-net
Generally speaking, trunking on an end-device is not a great idea, so perhaps, depending on your situation, it may be worth exploring purchasing a second NIC and attaching it to a different sys-net vm. That way you’d be effectively separating VLAN traffic into two different vms (sys-net-10 and sys-net-20).
Yes, although in this model sys-net wouldn’t be just an “end-device” as it would provide vlans similar to how the switch is doing it (sys-firewall* would deal only with untagged traffic). For a 2nd nic, I already tested with an external adapter and it worked, however I have way more than 2 VLANs to configure in sys-net (diagram was just a simple example), so can’t go using that route.
Just covering the basics here, sorry if it’s obvious: he indicates to place scripts in /rw/config/rc.local and /rw/config/qubes-firewall-user-script. If your sys-net is a DispVM you’ll need to edit the dvm-template which sys-net is based on, otherwise they won’t be there after a reboot.
Seems I was using sys-net vif ip address instead of sys-firewall eth0 ip address, but I also see that I need to “disconnect” from untagged connection. After doing these 2 changes things work correctly on sys-firewall and its connected appvms.
Will play a bit more and provide some documentation for others in need of a similar setup.
Looking for the promised documentation…
Can you post your iptables?
I have a hybrid port (not trunk or access) with only two vlans: 2 and 3, both tagged.
Both vlans come up. Called “02-mgmt” and “03-general”.
I have created a firewall-02, for the “02-mgmt” vlan.
On /sys-net/ I am trying to create rules. All seem to just turn into 0.0.0/0 for source and destination, no matter what I do.
Firgured out the issue. iptables -L for some reason always confuses. iptables -vL confirms that the settings were being made. I now have one AppVM with a custom firewall VM which is on the desired vlan, everything else on the “general” vlan.
bash-5.2# iptables -vL FORWARD
Chain FORWARD (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
18182 1654K ACCEPT all -- any 02-mgmt 10.138.10.236 anywhere
189 15206 DROP all -- any any 10.138.10.236 anywhere
428 22788 DROP all -- any 02-mgmt anywhere anywhere
0 0 DROP all -- any any anywhere anywhere state INVALID
18265 2582K ACCEPT all -- any any anywhere anywhere ctstate RELATED,ESTABLISHED
77 6158 QBS-FORWARD all -- any any anywhere anywhere
0 0 DROP all -- vif+ vif+ anywhere anywhere
77 6158 ACCEPT all -- vif+ any anywhere anywhere
0 0 DROP all -- any any anywhere anywhere
bash-5.2#
This is the vlan 02 interface:
bash-5.2# ip -4 addr show 02-mgmt
3: 02-mgmt@ens6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
inet 192.168.2.12/24 brd 192.168.2.255 scope global noprefixroute 02-mgmt
valid_lft forever preferred_lft forever
The 10.138.10.236 is the address assigned to the sys-firewall-02-mgmt VM.
Hello @QubesAnon, I actually dropped this and ended up doing some refactoring on VLANs and working with 2 physical nics instead, but glad you figured it out based on unman post.
Hi @equbes. I ended up with something different once I got things fully working…
There were issues regarding default routes being added to both VLANs: had to address that, so I ended up with some nmcli edits and some nm-gui additions and a somewhat simplified set of rules. Putting ip route commands in the /rw/config/rc.local did not work – something flushes all the routes and just added back the dual default routes, so NetworkMunger was where this was addressed.
I also needed an explicit route to the 192.168.1.0/24 vlan1. The vlan2 firewall-attached VMs are supposed to perform management functions on the vlan1-resident infrastructure. I have one VM which performs the management stuff, and the rest of the VMs are supposed to use vlan3. My firewall enforces those access restrictions. The Qubes issues arise because I wish to have VM attach to vlan2 for management and the rest vlan3 for “normal” access. The management VM also can access all the stuff on vlan3.
bash-5.2# iptables -vL LOGGING
Chain LOGGING (2 references)
pkts bytes target prot opt in out source destination
206 37507 LOG all -- any any anywhere anywhere LOG level warning prefix "ipt denied: "
206 37507 DROP all -- any any anywhere anywhere
bash-5.2# iptables -vL FORWARD
Chain FORWARD (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
12711 1689K ACCEPT all -- any 02-mgmt 10.138.10.236 anywhere
75796 7876K ACCEPT all -- any 03-general anywhere anywhere
0 0 LOGGING all -- any any 10.138.10.236 anywhere
206 37507 LOGGING all -- any 02-mgmt anywhere anywhere
0 0 DROP all -- any any anywhere anywhere state INVALID
109K 90M ACCEPT all -- any any anywhere anywhere ctstate RELATED,ESTABLISHED
0 0 QBS-FORWARD all -- any any anywhere anywhere
0 0 DROP all -- vif+ vif+ anywhere anywhere
0 0 ACCEPT all -- vif+ any anywhere anywhere
0 0 DROP all -- any any anywhere anywhere
bash-5.2# ip -4 route
default via 192.168.3.1 dev 03-general proto dhcp src 192.168.3.12 metric 401
10.138.10.236 dev vif10.0 scope link metric 32742
10.138.31.72 dev vif9.0 scope link metric 32743
192.168.1.0/24 via 192.168.2.1 dev 02-mgmt proto static metric 400
192.168.2.0/24 dev 02-mgmt proto kernel scope link src 192.168.2.12 metric 400
192.168.2.1 dev 02-mgmt proto static scope link metric 400
192.168.3.0/24 dev 03-general proto kernel scope link src 192.168.3.12 metric 401
bash-5.2# ip -4 addr
3: 03-general@ens6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
inet 192.168.3.12/24 brd 192.168.3.255 scope global dynamic noprefixroute 03-general
valid_lft 79554sec preferred_lft 79554sec
4: 02-mgmt@ens6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
inet 192.168.2.12/24 brd 192.168.2.255 scope global dynamic noprefixroute 02-mgmt
valid_lft 79554sec preferred_lft 79554sec
5: vif9.0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 32
inet 10.137.0.6/32 scope global vif9.0
valid_lft forever preferred_lft forever
6: vif10.0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 32
inet 10.137.0.6/32 scope global vif10.0
valid_lft forever preferred_lft forever
bash-5.2#
Being able to isolate the traffic coming from AppVM1 and AppVM2, from outside of the computer, is important in my usecase, and the only I’ve done this is by using VPN, which is overkill IMO…
Hello everyone,
I’m also interested in being able to send a segmented (VLAN) flow in trunk mode and then split it by app or VM. Has anyone managed to make progress on this issue?