How to configure split tunneling in wireguard sys-vpn NetVM?

I have setup a sys-vpn Qubes using the community guide here. My sys-vpn template is Debian-12 with Wireguard installed and nothing else.

I use a Wireguard config from ProtonVPN, which has AllowedIPs = 0.0.0.0/0, ::/0 and some Endpoint = 1.2.3.4:51820 in the peer configuration. I added this Endpoint to the Qubes Firewall settings per the original guide and everything works flawlessly for WAN connections. By default, this prohibits direct access to LAN hosts.

I need to setup split tunneling (on sys-vpn, if I am not mistaken?), such that any AppVMs using sys-vpn as their NetVM can:

  • Ping 1.1.1.1 through the Wireguard interface.
  • Ping my local router gateway at 172.22.132.1 through eth0 interface, bypassing Wireguard.

I have tried to manually update some settings to be able to connect to 172.16.0.0/12 locally, and otherwise route through Wireguard.

What I have attempted:

  • Using this IP calculator to disallow 172.16.0.0/12, calculate the AllowedIPs = 0.0.0.0/1, 128.0.0.0/3, 160.0.0.0/5, 168.0.0.0/6, 172.0.0.0/12, 172.32.0.0/11, 172.64.0.0/10, 172.128.0.0/9, 173.0.0.0/8, 174.0.0.0/7, 176.0.0.0/4, 192.0.0.0/2
  • I manually updated the AllowedIPs (per above) in my wg0.conf file
  • Run nmcli connection delete wg0 and then nmcli connection import type wireguard file wg0.conf to re-import the updated Wireguard config
  • Add 172.16.0.0/12 to the allowed connections in the Qubes Firewall Rules for sys-vpn
  • Restart sys-vpn

My updated Wireguard config looks like this:

[Interface]
PrivateKey = [REDACTED]
Address = 10.2.0.2/32
DNS = 10.2.0.1

[Peer]
PublicKey =  [REDACTED]
#AllowedIPs = 0.0.0.0/0, ::/0
AllowedIPs = 0.0.0.0/1, 128.0.0.0/3, 160.0.0.0/5, 168.0.0.0/6, 172.0.0.0/12, 172.32.0.0/11, 172.64.0.0/10, 172.128.0.0/9, 173.0.0.0/8, 174.0.0.0/7, 176.0.0.0/4, 192.0.0.0/2
Endpoint =  [REDACTED]:51820

After taking the above steps, I am then able to ping my local router gateway at 172.22.132.1. But I can no longer establish WAN connections (ping 1.1.1.1 does not work).

So I am only able to either reach local networks, or WAN networks, and not both simultaneously via split tunneling.

I have not setup any special nft rules. I have not done any additional hardening steps discussed in this guide. I have tried all the same steps again, rebuilding sys-vpn from scratch and have the same result.

Is there something else I need to do? Should I expect this to work under any circumstance? Finally, is there any reason I should not try to do this? Assuming either that I do or do not completely trust the hosts in 172.16.0.0/12.

In my ivpn guide, I remember I wrote something to allow bypassing the vpn for a few addresses, this does not work from the vpn qube itself though, but only in qubes connected to it.

In IVPN App 4.2 setup guide see Access local LAN / addresses

With this setup, you do not need to modify the vpn configuration, the bypass is handled by nftables.

How should I adapt the rc.local script for Wireguard? For example, the referred guide says to add the following to rc.local, but I do not have a /opt/ivpn/ directory:

if ! grep "QUBES OS" /opt/ivpn/etc/firewall.sh >/dev/null
then
    sudo sed -i '/-set_dns/a\
      #QUBES OS - specific operation\
      systemctl restart systemd-resolved || echo "Error: systemd-resolved" # this line is required for Qubes OS 4.2 (tested on Qubes OS 4.2-RC4)\
      /usr/lib/qubes/qubes-setup-dnat-to-ns || echo "Error: failed to run /usr/lib/qubes/qubes-setup-dnat-to-ns"' /opt/ivpn/etc/firewall.sh
      /rw/config/bypass-fw
fi

Could you connect the VPN, type the commands and see if you can reach your LAN from a qube using the VPN as its netvm? If it works we can find the best way to implement it automatically at boot :slight_smile:

Adding them in the qubes firewall script should do the trick anyway, but try manually first please.

I do not have the systemd-resolved service.

After executing /usr/lib/qubes/qubes-setup-dnat-to-ns, I still cannot ping my local router gateway:

user@sys-vpn:~$ ping -c 3 1.1.1.1
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=54 time=31.6 ms
64 bytes from 1.1.1.1: icmp_seq=2 ttl=54 time=36.1 ms
64 bytes from 1.1.1.1: icmp_seq=3 ttl=54 time=30.4 ms

--- 1.1.1.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 30.362/32.667/36.068/2.454 ms
user@sys-vpn:~$ 
user@sys-vpn:~$ 
user@sys-vpn:~$ ping -c 3 172.22.132.1
PING 172.22.132.1 (172.22.132.1) 56(84) bytes of data.

--- 172.22.132.1 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2072ms

user@sys-vpn:~$ 
user@sys-vpn:~$ 
user@sys-vpn:~$ systemctl restart systemd-resolved
Failed to restart systemd-resolved.service: Unit systemd-resolved.service not found.
user@sys-vpn:~$ 
user@sys-vpn:~$ 
user@sys-vpn:~$ sudo systemctl restart systemd-resolved
Failed to restart systemd-resolved.service: Unit systemd-resolved.service not found.
user@sys-vpn:~$ 
user@sys-vpn:~$ 
user@sys-vpn:~$ /usr/lib/qubes/qubes-setup-dnat-to-ns 
Traceback (most recent call last):
  File "/usr/lib/qubes/qubes-setup-dnat-to-ns", line 144, in <module>
    install_firewall_rules(get_dns_resolved())
  File "/usr/lib/qubes/qubes-setup-dnat-to-ns", line 132, in install_firewall_rules
    old_rules = subprocess.check_output(
                ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/subprocess.py", line 466, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/subprocess.py", line 548, in run
    with Popen(*popenargs, **kwargs) as process:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/subprocess.py", line 1024, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.11/subprocess.py", line 1901, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'nft'
user@sys-vpn:~$ 
user@sys-vpn:~$ 
user@sys-vpn:~$ sudo /usr/lib/qubes/qubes-setup-dnat-to-ns 
user@sys-vpn:~$ 
user@sys-vpn:~$ 
user@sys-vpn:~$ ping -c 3 1.1.1.1
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=54 time=229 ms
64 bytes from 1.1.1.1: icmp_seq=2 ttl=54 time=170 ms
64 bytes from 1.1.1.1: icmp_seq=3 ttl=54 time=40.5 ms

--- 1.1.1.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 40.501/146.556/229.408/78.846 ms
user@sys-vpn:~$ 
user@sys-vpn:~$ 
user@sys-vpn:~$ ping -c 3 172.22.132.1
PING 172.22.132.1 (172.22.132.1) 56(84) bytes of data.

--- 172.22.132.1 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2088ms

Sorry if I wasn’t clear enough, I didn’t have much time to write more detailed explanations.

The important code here is the following:

#!/bin/sh

IPS_TO_ALLOW="10.42.42.42 10.42.42.150"

for i in "$IPS_TO_ALLOW"
do
  nft insert rule qubes custom-forward ip daddr $i counter accept
  nft insert rule filter FORWARD ip daddr $i counter accept
  nft insert rule filter FORWARD ip saddr $i counter accept
done

Adapt to your ip addresses you need, I didn’t try but it should work with a subnet too.

The script you tried to run is to handle the dns redirection, this is not needed here, only this snippet of shell that allows traffic over the interface and forward before the vpn for the subset of addresses.

I updated the script as follows (sudo is necessary, otherwise I get nft: not found error):

#!/bin/sh

IPS_TO_ALLOW="172.22.132.1 172.22.132.175"

for i in "$IPS_TO_ALLOW"
do
  sudo nft insert rule qubes custom-forward ip daddr $i counter accept
  sudo nft insert rule filter FORWARD ip daddr $i counter accept
  sudo nft insert rule filter FORWARD ip saddr $i counter accept
done

I made the script executable and ran it, getting the following error:

Error: syntax error, unexpected counter
insert rule qubes custom-forward ip daddr 172.22.132.1 172.22.132.175 counter accept
                                                                      ^^^^^^^
Error: syntax error, unexpected counter
insert rule filter FORWARD ip daddr 172.22.132.1 172.22.132.175 counter accept
                                                                ^^^^^^^
Error: syntax error, unexpected counter
insert rule filter FORWARD ip saddr 172.22.132.1 172.22.132.175 counter accept
                                                                ^^^^^^^

The syntax is different if you put multiple addresses in the rule, the script iterates or each IP to write one rule.

See daddr examples on Quick reference-nftables in 10 minutes - nftables wiki

I literally took the code you provided, and replaced 10.42.42.42 10.42.42.150 (two IPs) with 172.22.132.1 172.22.132.175 (two IPs), and prepended the nft commands with sudo.

Nevertheless, I have updated the script as follows:

#!/bin/sh

IPS_TO_ALLOW="172.16.0.0/12"

sudo nft insert rule qubes custom-forward ip daddr "$IPS_TO_ALLOW" counter accept
sudo nft insert rule filter FORWARD ip daddr "$IPS_TO_ALLOW" counter accept
sudo nft insert rule filter FORWARD ip saddr "$IPS_TO_ALLOW" counter accept

Running the script raises these errors:

Error: Could not process rule: No such file or directory
insert rule filter FORWARD ip daddr 172.16.0.0/12 counter accept
            ^^^^^^
Error: Could not process rule: No such file or directory
insert rule filter FORWARD ip saddr 172.16.0.0/12 counter accept
            ^^^^^^

From the documentation you provided, the correct syntax is nft insert rule [<family>] <table> <chain> [position <handle>] <matches> <statements>.

filter is not a valid family.

family refers to a one of the following table types: ip, arp, ip6, bridge, inet, netdev. It defaults to ip.

Nor is it a valid table name:

user@sys-vpn:~$ sudo nft list tables
table ip qubes
table ip6 qubes
table ip qubes-firewall
table ip6 qubes-firewall
table inet qubes-nat-accel

Ok, I’m truly sorry, I remember that IVPN app was adding some nft rules. I’ll figure something that works on qubes os and come by with a working example that I tested before.

You do not need to be sorry about anything. I appreciate your continued support. Thank you :pray:

You can run this in the vpn qube, this creates a route that has a higher priority than WireGuard.

sudo ip route add 192.168.1.1/32 via 10.138.25.44 dev eth0

Where

  • 10.138.25.44 is the qube’s default route, which you can get with ip r | grep ^default | cut -d ' ' -f 3
  • 192.168.1.1/32 is a subnet / ip you want to reach without the VPN

This isn’t ideal, but it works for me, add this to /rw/config/rc.local

sleep 10
ip route add 192.168.1.1/32 via "$(ip r | grep ^default | cut -d ' ' -f 3)" dev eth0

Of course, you need to allow the subnet in the vpn qube’s firewall :wink:

1 Like

Works! Thank you so much for your effort and patience. :heart:

1 Like