Wireguard VPN setup

[apparatus]

@wilsonb

1. The described nft rule in /rw/config/qubes-firewall-user-script does not work automatically in my case. I made the script executable, but it still does not load when the VM starts. I confirmed this by running nft list table ip qubes. The custom-forward chain is empty. Only by running the script manually does the custom forward chain show the rule.

I am by no means a QubesOS or nftables expert. I may have misconfigured my system or run the checks incorrectly. Please help. My qube is based on Fedora 40 Minimal.

Is your qube a service qube with “Provides network” option enabled?
The qubes-firewall-user-script script is only executed for the qubes with qubes-firewall service enabled (it’s enabled by default for qubes with “Provides network” option enabled).

@wilsonb

2. After manually adding the nft rule, DuckDuckGo (and others) still won’t load properly. The ip link show eth0 command still shows an MTU of 1500. Again, I may have checked the wrong thing.

A second assumption: because I have a second VPN on my router, the nft rule is not reducing the MTU to a working size. I would appreciate some clarification on the nft rule. I am new to nftables.

I guess you need to configure the TCP MSS Clamping in your router as well.

[wilsonb]

@apparatus

Is your qube a service qube with “Provides network” option enabled?
The qubes-firewall-user-script script is only executed for the qubes with qubes-firewall service enabled (it’s enabled by default for qubes with “Provides network” option enabled).

1. I was not aware that I needed to enable the qubes-firewall service. Thanks for the info! I can confirm that after adding the service, the firewall command is now listed via nft list table ip qubes. This might be an interesting detail for @solene to add. The problem here might be that this is enabled by default, but not for VMs based on minimal templates. Not sure about that.

2. I thought that the firewall command had to be in every VM that I wanted to use my sys-vpn service VM with. So I set the firewall script in the VM where I launch my browser, NOT in the VM that has the Wireguard connection (sys-vpn). Is this wrong? I don’t understand why I should set my browser VM to provide the network.

My connection looks like this: browserVM ↔ sys-vpn ↔ sys-firewall ↔ sys-net. I have the firewall service/nft rule set in the browserVM. Does it belong in sys-vpn itself instead? Confused about this.

I set my own MTU fix for each VM I use to connect to a sys-vpn VM, such as my browserVM.

@apparatus

I guess you need to configure the TCP MSS Clamping in your router as well.

3. The router has the MTU set accordingly. My assumption here is that my QubesOS VM clamps to an optimal like 1500 - WireGuard overhead. It does not know that there is a second overhead due to the router. I am just speculating, I am not an expert on this. The interesting part that I can confirm is that my suggested MTU fix works fine.

Thanks for your time and help!

[apparatus]

@wilsonb

1. I was not aware that I needed to enable the qubes-firewall service. Thanks for the info! I can confirm that after adding the service, the firewall command is now listed via nft list table ip qubes. This might be an interesting detail for @solene to add. The problem here might be that this is enabled by default, but not for VMs based on minimal templates. Not sure about that.
2. I thought that the firewall command had to be in every VM that I wanted to use my sys-vpn service VM with. So I set the firewall script in the VM where I launch my browser, NOT in the VM that has the Wireguard connection (sys-vpn). Is this wrong? I don’t understand why I should set my browser VM to provide the network.

My connection looks like this: browserVM ↔ sys-vpn ↔ sys-firewall ↔ sys-net. I have the firewall service/nft rule set in the browserVM. Does it belong in sys-vpn itself instead? Confused about this.

I set my own MTU fix for each VM I use to connect to a sys-vpn VM, such as my browserVM.

You need to add this firewall rule just in the sys-vpn, not in every qube connected to sys-vpn.

@wilsonb

The router has the MTU set accordingly. My assumption here is that my QubesOS VM clamps to an optimal like 1500 - WireGuard overhead. It does not know that there is a second overhead due to the router. I am just speculating, I am not an expert on this. The interesting part that I can confirm is that my suggested MTU fix works fine.

If Path MTU Discovery is not working correctly between your router and sys-vpn then you need to clamp the MSS in your router as well.

[wilsonb]

Thanks again for your time. Two new discoveries:

1. I set the firewall rule in sys-vpn, but the service was still not enabled by default (maybe because it is based on fedora 40 minimal). qvm-service sys-vpn qubes-firewall on solved the problem.

2. Even with the firewall rule enabled (and confirmed with ntf list table ip qubes it shows up in the custom-forward chain) my browserVM1 does not work. My second browserVM2 with my fix and based on the same sys-vpn VM works fine. browserVM1 also shows MTU 1500 with ip link show eth0. What am I doing wrong?

Solene’s setup is strictly supiror to mine if it works, because I would not need to set the MTU fix in each browserVM (sys-vpn connected VM).

How can I continue troubleshooting? Have you tried Fedora Minimal or are you running the bloated Fedora-40-XFCE VM? Maybe I am missing a package?

[apparatus]

@wilsonb

I set the firewall rule in sys-vpn, but the service was still not enabled by default (maybe because it is based on fedora 40 minimal). qvm-service sys-vpn qubes-firewall on solved the problem.

It should be enabled by default if “Provides network” option is set:

qubes-firewall

Default: enabled only in ProxyVM

qvm-service – Manage (Qubes-specific) services started in VM — Qubes Admin client v4.3.5-0-g30d7547-dirty documentation

Even if it’s not listed in the Settings → Services tab it still should be enabled in the qube.

@wilsonb

1. Even with the firewall rule enabled (and confirmed with ntf list table ip qubes it shows up in the custom-forward chain) my browserVM1 does not work. My second browserVM2 with my fix and based on the same sys-vpn VM works fine. browserVM1 also shows MTU 1500 with ip link show eth0. What am I doing wrong?

Solene’s setup is strictly supiror to mine if it works, because I would not need to set the MTU fix in each browserVM (sys-vpn connected VM).

How can I continue troubleshooting? Have you tried Fedora Minimal or are you running the bloated Fedora-40-XFCE VM? Maybe I am missing a package?

Maybe this:

@apparatus

If Path MTU Discovery is not working correctly between your router and sys-vpn then you need to clamp the MSS in your router as well.

You can use tcpdump in the qube connected to sys-vpn to see which MSS is used:
Run tcpdump:

sudo tcpdump

Run curl to get some website:

curl https://debian.org

Check the output of tcpdump:

IP X.X.X.X.X > 151.101.194.132.https: Flags [S], seq X, win 64240, options [mss 1460,nop,nop,sackOK,nop,wscale 6], length 0
IP 151.101.194.132.https > X.X.X.X.X: Flags [S.], seq X, ack X, win 65535, options [mss 1412,nop,nop,sackOK,nop,wscale 9], length 0

[wilsonb]

Thanks again for your help! So I did what you said with a VM set up according to this guide and then once with my 1280 MTU fix. The results are as follows:

• Solene guide: 1460 MSS (first line), 1380 MSS (second line)
• My 1280 MTU fix: 1240 MSS (first line), 1380 MSS (second line)

What is the difference between the first row and the second row?

What is the interpretation of the result?

[apparatus]

Did you check it in the qube connected to sys-vpn?
Check it in the qube connected to the net qube of sys-vpn (e.g. sys-firewall) to see what MSS will be reported there.
What website is broken for you when you don’t use your ip link set dev eth0 mtu 1280 MTU fix?
Try to curl this website instead of debian.org.

[wilsonb]

I tested in my browserVM which is connected to sys-vpn. Is this correct?

Basically every page is broken. Google works for some reason. DuckDuckGo.com, Speedtest.net and many others (all I tested) don’t work. Well, they partially load and break. So they work half.

[apparatus]

@wilsonb

I tested in my browserVM which is connected to sys-vpn. Is this correct?

Yes.
Check the same with browserVM connected to the net qube of sys-vpn (e.g. sys-firewall) and see what MSS will be there.

[wilsonb]

02:47:29.441574 IP TEMPNET.59712 > 151.101.130.132.http: Flags [S], seq 552503776, win 64240, options [mss 1460,nop,nop,sackOK,nop,wscale 6], length 0
02:47:29.624672 IP 151.101.130.132.http > TEMPNET.59712: Flags [S.], seq 3150086529, ack 552503777, win 65535, options [mss 1240,nop,nop,sackOK,nop,wscale 9], length 0

This is my browserVM (TMPNET) connecting directly to sys-firewall (hence skipping sys-vpn)

What is the interpretation of the result?

[apparatus]

Did you remove your MTU fix ip link set dev eth0 mtu 1280 from browserVM?

[wilsonb]

Yes!

3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group 1 qlen 1000

Do you want the same result with the MTU 1280 fix?

[apparatus]

@wilsonb

Do you want the same result with the MTU 1280 fix?

No.

Do you have this MTU fix in your sys-firewall/sys-net or in router?
For some reason your working MSS is reported to be 1240.

[wilsonb]

Yes, I am very confused about this.

I am using pfSense and my VPN interface settings are: MTU 1420, MSS 1280.

When I set my MTU 1280 fix in my browserVM everything works fine.

With Solene’s MSS clamp in sys-vpn nothing really works. The pages load and then break.

[apparatus]

@wilsonb

I am using pfSense and my VPN interface settings are: MTU 1420, MSS 1280.

I don’t know why is your MSS set to 1280 in the router. Did you set it manually or is it set in the VPN config in the router?
In any case, the proper way would be to add the TCP MSS clamp in your router config as well.

[wilsonb]

I am new to this and followed my VPN provider’s recommendation for pfSense. What is your suggestion?

[apparatus]

This:

@apparatus

In any case, the proper way would be to add the TCP MSS clamp in your router config as well.

[wilsonb]

So what that means? What settings?

[apparatus]

I don’t have pfSense so no idea. If it has nftables then add the same nftables rule as in the guide.

[wilsonb]

If I can’t adjust the settings. What are the recommended values?