I would like to create my own custom firewall VM without using the qubes-firewall implementation, meaning that I want to create a netvm that all VMs connect to and then there I want to setup my own nftables configuration.
When I have a netvm that services several other VMs, then in the netvm I have several nic’s, one for each client VM, example:
user@netvm:~$ ip a
[...]
13: vif19.0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group 2 qlen 1000
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
inet 10.138.30.88/32 scope global vif19.0
valid_lft forever preferred_lft forever
inet6 fe80::fcff:ffff:feff:ffff/64 scope link
valid_lft forever preferred_lft forever
15: vif23.0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group 2 qlen 1000
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
inet 10.138.30.88/32 scope global vif23.0
valid_lft forever preferred_lft forever
inet6 fe80::fcff:ffff:feff:ffff/64 scope link
valid_lft forever preferred_lft forever
[...]
I want to:
associate those nic’s to the client VMs (if that information is only available in dom0 thats ok)
Also: when I run nft list chains on a netvm, it shows me:
table ip qubes-firewall {
chain forward {
type filter hook forward priority filter; policy drop;
}
chain prerouting {
type filter hook prerouting priority raw; policy accept;
}
chain postrouting {
type filter hook postrouting priority raw; policy accept;
}
chain qbs-10-137-0-45 {
}
[... lots more of those qbs-<IP> ...]
}
chain qbs-10-138-10-201 {
}
These seem to match the IPs of the nic’s in the client VMs. What causes these to be generated and how does it work?
My goal is to write nftables rules like (pseudocode) “if from interface vif5.0 and dest is internet allow”, but of course I need to trigger some management code for those as soon as the client boots.
In NetVMs/ProxyVMs, scripts placed in /rw/config/network-hooks.d will be ran when configuring Qubes interfaces. For each script, the command, vif, vif_type and ip is passed as arguments (see /etc/xen/scripts/vif-route-qubes). For example, consider a PV app qube work with IP 10.137.0.100 and sys-firewall as NetVM. Assuming it’s Xen domain id is arbitrary 12 then, the following script located at /rw/config/network-hooks.d/hook-100.sh in sys-firewall:
#!/bin/bash
command="$1"
vif="$2"
vif_type="$3"
ip="$4"
if [ "$ip" == '10.137.0.100' ]; then
case "$command" in
online)
ip route add 192.168.0.100 via 10.137.0.100
;;
offline)
ip route del 192.168.0.100
;;
esac
fi
will be executed with arguments online vif12.0 vif 10.137.0.100 when starting work. Please note that in case of an HVM, the script will be called twice - once with vif_type vif, then with vif_type vif_ioemu (and different interface names). As long as the ioemu interface exists, it should be preferred (up to the hook script). When the VM decides to use a PV interface (vif_type vif), the ioemu one will be unplugged.
You can create a script in dom0 that will create the associated list “IP address - qube name” and pass it to your custom firewall VM.
You can also add a custom Qubes RPC service that could be called from sys-custom-fw qube script in /rw/config/network-hooks.d on new qube connect/disconnect event to request the dom0 to update the connected qubes associated list:
You can use QubesDB to pass the list to your custom firewall VM, e.g:
In dom0:
# get the list of IP addresses of qubes
qubesdb-list /connected_qubes_list/
# read the qube name associated with this IP address
qubesdb-read /connected_qubes_list/10.138.30.88
I am wondering where to find the templates(?) that create any cube’s nft rules, if netvm or just a cube thats connected to a netvm. When I run nft list ruleset in a netvm there are a good number of rules already - where do those come from?
When a netVM has qvm-service qubes-firewall true, then there are two extra chains ip qubes-firewall and ip6 qubes-firewall.
Those are the chains that the firewall gui sets, by default:
chain qbs-10-137-0-45 {
accept
reject with icmp admin-prohibited
}
when adding a rule with qvm-firewall from dom0 (just an example I know I didn’t delete the first rule)
chain qbs-10-137-0-51 {
accept
ip daddr 8.8.8.8 udp dport 4444 accept
reject with icmp admin-prohibited
}
I would like to replace the reject with icmp admin-prohibited with something like:
log prefix "QBS_DROP" level info
and then just drop it.
I think the reject with icmp admin-prohibitet rule is coming from sys-firewall:/lib/python3/dist-packages/qubesagent/firewall.py L490. I also didn’t find any “hook” script option or similar in firewall.py.
So I think if I want to turn:
chain qbs-10-137-0-51 {
ip daddr 8.8.8.8 udp dport 4444 accept
reject with icmp admin-prohibited
}
info
chain qbs-10-137-0-51 {
ip daddr 8.8.8.8 udp dport 4444 accept
log prefix "qbs-10-137-0-51_DROP" level info
drop
}
I could just “bash script iterate” all those qbs- chains and change this last line into my two last lines.
From what I can tell the qubes-firewall just loads all the rules, regardless if the VM is on or off. How does that work for disp0123 though, where the IP is not known before? What triggers qubes-firewall to create a new chain for that?
PS: What I actually want is to log all packages that the firewall VM blocked.
Replace the rule with handle 14 to add logging and a counter:
nft replace rule qubes-firewall qbs-10-138-26-180 handle 14 counter log prefix \"[qubes-firewall-BLOCKED] \" reject with icmp type admin-prohibited
My remaining question is whats the best way to automate that. When qubes-firewall creates a chain, I want to run a “post hook” script that configures this chain to not reject with admin-prohibited but to log & drop.
I guess you can watch the /qubes-firewall-handled/ QubesDB entry and add your custom rules on change.
From the /lib/python3/dist-packages/qubesagent/firewall.py L167:
def update_handled(self, addr):
"""
Update the QubesDB count of how often the given address was handled.
User applications may watch these paths for count increases to remain
up to date with QubesDB changes.
"""
cnt = self.qdb.read('/qubes-firewall-handled/{}'.format(addr))
try:
cnt = int(cnt)
except (TypeError, ValueError):
cnt = 0
self.qdb.write('/qubes-firewall-handled/{}'.format(addr), str(cnt+1))
You can watch the path in QubesDB in the similar manner:
From the /lib/python3/dist-packages/qubesagent/firewall.py L341:
try:
for watch_path in iter(self.qdb.read_watch, None):
if watch_path == '/connected-ips':
self.update_connected_ips(4)
if watch_path == '/connected-ips6':
self.update_connected_ips(6)
# ignore writing rules itself - wait for final write at
# source_addr level empty write (/qubes-firewall/SOURCE_ADDR)
if watch_path.startswith('/qubes-firewall/') and watch_path.count('/') == 2:
source_addr = watch_path.split('/')[2]
self.handle_addr(source_addr)
except OSError: # EINTR
# signal received, don't continue the loop
pass
There is also qubesdb-watch CLI tool if you want to use it in bash scripts.
