Wireguard VPN setup

apparatus · October 8, 2024, 9:09am

You mean what is a recommended MSS value for the VPN in the router config? I guess if your VPN provider tells you to use 1280 then use it.

wilsonb · October 8, 2024, 9:10am

Okay, I will sleep on it and get back to you. Thanks for today. Still confused… Networking is fun.

apparatus · October 8, 2024, 9:20am

Also note that this only works for TCP, if you connect to your VPN in sys-vpn using UDP then I think it won’t work.
In case of UDP you’ll need to set the MTU value to 1280 in the VPN config in sys-vpn.

wilsonb · October 10, 2024, 1:41am

I did some testing today. This is the current state:

I checked the videos of the maintainer/dev of the pfSense WireGuard package (Christian McDonald). He sets the MTU/MSS in pfSense to 1420, both with some vague comments that make me think he is not entirely sure either. I have tested with 1420/1420 on my pfSense router, and it seems to work for sys-firewall connected VMs.

Note that setting 1420/1420 in pfSense does not mean that these values will be used. pfSense subtracts 40 and 60 for IPv4 and IPv6.

The description of the MSS field in pfSense: “If a value is entered in this field, then MSS clamping for TCP connections to the value entered above minus 40 for IPv4 (TCP/IPv4 header size) and minus 60 for IPv6 (TCP/IPv6 header size) will be in effect.”

Here the video: https://youtu.be/wYe7FzZ_0X8?feature=shared&t=785

I tested with these values in a VM connected to sys-firewall. Everything works, speeds are great!
I tested my sys-vpn with Solene’s MTU fix, nothing works. Do I need to set the same fix in sys-firewall to reduce packet size twice due to the dual VPN setup?

apparatus · October 11, 2024, 8:37pm

Add MTU = 1340 in the wireguard config in sys-vpn and add the TCP MSS clamp nft rule in sys-vpn.
If you don’t use IPv6 then you can set MTU=1440 in router and MTU=1380 in sys-vpn.

wilsonb · October 12, 2024, 5:37am

I appreciate your time and help. However, I am not quite sure what you mean here and need some additional help.

Current state

My working (qubes firewall service enabled) sys-vpn currently has Solene’s firewall line: nft add rule ip qubes custom-forward tcp flags syn / syn,rst tcp option maxseg size set rt mtu.
I can confirm that the firewall command is loaded as I see it in the custom-forward chain.

Why is that? Because the WireGuard protocol itself has a 60-byte overhead when considering IPv4?

Questions

Why should I set your stated MTU/MSS values (please walk me through the calculation and reasoning)?
How should I set the MTU in sys-vpn?
In general, I would appreciate an explanation of MTU/MSS. It seems quite confusing. My understanding is that normal links have a size of 1500 bytes. To avoid fragmentation, this needs to be lowered when additional protocols like WireGuard come into play. What is the difference between MTU and MSS? I know the words, but what is the effect?

Is it that the MSS is the payload that I subtract from the link size to get the MTU? So the MTU is the resulting link size when I subtract the overhead expressed by the MSS?

My Research

The WireGuard MSS overhead consists of:

UDP header (8 bytes): WireGuard uses UDP as its transport protocol, which adds a 8-byte header to each packet.
WireGuard header (20 bytes for IPv4, 40 bytes for IPv6): This header contains metadata such as packet sequence numbers, authentication tags, and other control information. For IPv4, the header is 20 bytes long, while for IPv6, it’s 40 bytes long.
Payload overhead (8 bytes for IPv4, 16 bytes for IPv6): This includes additional bytes required for packet fragmentation and reassembly, as well as other protocol-specific fields.

The total WireGuard MSS overhead is:

For IPv4: 20 (WireGuard header) + 8 (payload overhead) = 28 bytes
For IPv6: 40 (WireGuard header) + 16 (payload overhead) = 56 bytes

To calculate the effective MTU (Maximum Transmission Unit) for WireGuard, you need to subtract this overhead from the underlying link MTU. For example:

If the underlying link MTU is 1500 bytes, and you’re using IPv4, the effective WireGuard MTU would be: 1500 - 28 = 1472 bytes
If the underlying link MTU is 1500 bytes, and you’re using IPv6, the effective WireGuard MTU would be: 1500 - 56 = 1444 bytes

Is my research correct? Please help me. This is confusing and difficult to understand.

If I am correct so far, is there any reason to use a smaller number? I have never seen anyone use 1444 bytes, but values like 1440. Why is that? Just because it is more convenient and nobody cares about the 4 bytes or is there something else?

Let’s also assume that I want to use IPv6, or at least keep the option. Do I put 1440 bytes in pfSense? I ask because the MSS field has the following description:

If a value is entered in this field, then MSS clamping for TCP connections to the value entered above minus 40 for IPv4 (TCP/IPv4 header size) and minus 60 for IPv6 (TCP/IPv6 header size) will be in effect.

My understanding (and I may be completely wrong here) is that I only set the link size - WireGuard payload, as this is the overhead that the WireGuard protocol requires as an extra. Since the connection itself is TCP/IPv4 or TCP/IPv6, pfSense takes care of the rest. But why? Why should I wrap the WireGuard UDP in IPv4/6 TCP so that I have to subtract an extra 40/60 bytes? Is that what is happening here?

While I am quite confused about this, I would assume that if I go for IPv6 (which then also works for IPv4) and set 1440 in the router, then the real MTU is 1380 (since pfSense subtracts additional 60 bytes). So should 1380 be my sys-vpn link size? What about the second VPN payload/overhead?

Since there are many questions, I will not speculate further, but let you explain first. Thank you very much! Setting up a VPN in Qubes is an interesting journey through networking.

I have also often seen the recommendation for MTU 1420, MSS 1380. Why is that? 1420 because of the WireGuard overhead? What is the difference between MTU and MSS?

apparatus · October 12, 2024, 10:12am

MTU overhead

The overhead of WireGuard breaks down as follows:[15]

20-byte IPv4 header or 40 bytes IPv6 header

8-byte UDP header

4-byte type

4-byte key index

8-byte nonce

N-byte encrypted data

16-byte authentication tag

WireGuard - Wikipedia

So the overhead for IPv4 is 60 bytes and for IPv6 is 80 bytes.
For your router it’d be 1500-60=1440
For sys-vpn 1440-60=1380

Set it in the wireguard config e.g.:

[Interface]
Address = X.X.X.X/32
PrivateKey = X
DNS = X.X.X.X
MTU = 1380
[Peer]
PublicKey = X
PresharedKey = X
Endpoint = X.X.X.X:X
AllowedIPs = 0.0.0.0/0

Maximum segment size - Wikipedia

If you set the MTU in sys-vpn wireguard config to 1380 then the TCP MSS clamp nft rule will indicate the qubes connected to sys-vpn that MSS is limited to 1340 (MTU 1380 - 20 (IPv4 header) - 20 (TCP overhead)).
This clamp is only for TCP and won’t work for other protocols e.g. UDP/ICMP so they will be fragmented. E.g. if send a UDP packet with the size > 1352 (MTU 1380 - 20 (IPv4 header) - 8 (UDP header)) from the qube connected to sys-vpn then it’ll be sent by sys-vpn as 2 or more UDP packets.

barto · October 13, 2024, 8:46am

FWIW, the native wireguard client(**) sets the MTU of the (wireguard) tun device to 1420.
** This statement is true for the latest MacOS wireguard client (v1.0.16) as well as the latest Debian wireguard client (v.1.0.20).

There’s a lot of confusion around MTU and MSS, as well as a mix between encrypted tunneled traffic and the wireguard “carrier” udp packets.

I don’t have time to go into details now and will try to detail later in the week.

apparatus · October 13, 2024, 8:50am

The default is 1420 MTU to support both IPv4 and IPv6, but it you don’t use IPv6 then you can set 1440 MTU to have more throughput.

MTU operational considerations

Assuming the underlay network transporting the WireGuard packets maintains a 1500 bytes MTU, configuring the WireGuard interface to 1420 bytes MTU for all involved peers is ideal for transporting IPv6 + IPv4 traffic. However, when exclusively carrying legacy IPv4 traffic, a higher MTU of 1440 bytes for the WireGuard interface suffices.[14]

From an operational perspective and for network configuration uniformity, choosing to configure a 1420 MTU network-wide for the WireGuard interfaces would be advantageous. This approach ensures consistency and facilitates a smoother transition to enabling IPv6 for the WireGuard peers and interfaces in the future.

WireGuard - Wikipedia

solene · October 13, 2024, 8:58am

As Qubes OS adds another NAT, it make sense to me that the MTU should not be the same as on a LAN, when using Qubes OS.

apparatus · October 13, 2024, 9:07am

NAT is not adding overhead so it’s not changing MTU. Various tunnels that add overhead will affect MTU.

wilsonb · October 22, 2024, 7:34am

Much appreciated!

So how do I avoid fragmentation of non-TCP packets? Internal QubesOS is all TCP, right? So it should not be a problem here, but I would like to know how to achieve this.

Thanks!

wilsonb · October 22, 2024, 7:36am

That’s my specialty. LOL.

wilsonb · October 22, 2024, 8:34am

Update on my double VPN issue:

Set MTU and MSS fields in pfSense to 1440 (IPv4 only)
Confirm settings with Diagnostics > Command Prompt ifconfig tun_wg0 that MTU is 1440
Set WireGuard configuration with nmcli c (list connections); nmcli c edit vpn-connection-name; set wireguard.mtu 1380 (set WireGuard MTU to 1380 (IPv4 only)); print wireguard.mtu; save; quit (I guess I also reloaded the tunnel)
Confirm the WireGuard interface with ip link show. Shows mtu 1380
Reboot sys-vpn and browserVM (sys-vpn connected VM)
Nothing works. Well, Google loads fast, DuckDuckGo and others not.
Did the same with 1420 for MTU + MSS on pfSense and 1340 for sys-vpn. It works damn well! WHY??? It works! But why? Does Qubes use IPv6 internally?
I will test my setup and come back in a few days to really confirm. But this is the first time I can immediately open DuckDuckGo with a clean Brave browser or run fast speed tests.

WHY? Please explain! MTU/MSS is confusing as it never works as explained! Why?

apparatus · October 22, 2024, 10:25am

By setting the correct MTU for the links.

What do you mean by “Internal QubesOS”?
The qubes are connected as normal network nodes, then can use TCP/UDP/ICMP/etc to communicate.

I don’t know why is it 1340 if you set it to 1380. I didn’t try to use nmcli so maybe it’s some NetworkManager peculiarity, but if you set MTU = 1380 in the wireguard config file then the wireguard network interface will have MTU = 1380 in the output of ip link show.

Check the output of ip a in the sys-net terminal, maybe your pfSense is using IPv6 for LAN.

wilsonb · October 23, 2024, 5:24am

Oh, sorry, that was a documentation error on my part!

What should I check here?

wilsonb · October 23, 2024, 5:28am

After testing my VPN connections yesterday and being very happy with them, I ran into a problem this morning that cost me 1 hour of my life just to get Tor working again. For some strange reason, sys-whonix will not bootstrap (work) if my MTU and MSS are set to 1420 on pfSense. I then lowered the MSS significantly and sys-whonix worked again. Why is this happening? I know the MTU/MSS settings are the important part, but what are the correct settings?

apparatus · October 23, 2024, 7:13am

Check the IP addresses assigned yo your network interface that is connected to pfSense and make sure that IPv6 address is not assigned to it.

I don’t know what and how you configured in pfSense, but with MTU=1420 your MSS should be 1380 for IPv4.

Is sys-whonix connected to sys-firewall/sys-net or to sys-vpn?
Did you add TCP MSS clump firewall rule in pfSense config?
Did you block ICMP between sys-whonix and pfSense?
But I’m not sure why lowering the MSS in pfSense would make sys-whonix work.

Alethia · October 29, 2024, 9:51am

I have issues getting this to work on Fedora 40. I’ve tested Fedora 40, Fedora 40 xfce and Fedora 40 minimal. For minimal I installed the following packages:

sudo dnf install qubes-core-agent-passwordless-root qubes-core-agent-nautilus qubes-menus nautilus qubes-core-agent-networking nano gedit wireguard-tools qubes-core-agent-network-manager qubes-core-agent-dom0-updates network-manager-applet notification-daemon

If I instead use Fedora 38 minimal with the above packages (I just change template VM for my sys-vpn) then sys-vpn works. I’ll happily continue with 38 for some time but am curious as to what I might be missing. I did follow the rest of the steps in the guide:

Set “provides network” on sys-vpn
Enabled service “network-manager” on sys-vpn
Use Qube GUI to set firewall to the VPN endpoint
Connection is setup with nmcli
Added nft rules for mtu and hardening

Using any Fedora 40 for sys-vpn and testing curl I get the following result:

[user@sys-vpn ~]$ curl google.com
curl: (6) Could not resolve host: google.com
[user@sys-vpn ~]$ curl 194.68.59.6
curl: (7) Failed to connect to 194.68.59.6 port 80 after 0 ms: Couldn't connect to server

With Fedora 38 it looks like this:

[user@sys-vpn ~]$ curl google.com
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">here</A>.
</BODY></HTML>
[user@sys-vpn ~]$ curl 194.68.59.6
<html>Nginx is functioning normally</html>

What could I be missing?

solene · October 29, 2024, 10:41am

if you type sudo wg, in the peer information, do you have a value different than 0 in the received data?