Troubleshooting solution for minimal VPN qube(s) on 4.2

[eBread]

Hello, it has come to my attention that many users moving towards 4.2 had issues with their 4.1 VPN qube upgraded or newly created qube(s). After some troubleshooting I thought I’d share what worked and what didn’t.

I followed @Pawelek85 's guide and the one created by @solene here. Having prior knowledge on the topic as I created all of my sys-* qubes on 4.2 based on minimals I installed the required packages (will come to that later).

After some troubleshooting I had 3 vpn templates, one based on debian-12-xfce working great and two cloned from debian-12-minimal with different packages. One of the minimal vpn template had a lot of remaining junk after I tried multiple packages but that didn’t affect it since I had the vpn qube working, however no connection was made from the AppVM connected to it.

My setup usually includes a vpn firewall ( AppVM → VPN qube → vpn qube firewall → sys-firewall) to manage the authorized IPs the VPN can reach, to make sure if the VPN gets compromised somehow, it still cannot reach anything besides what the firewall allows it to access (thanks to qvm-firewall configurations made from dom0)

From what I could observe, nftables may or may not be the culprit. What I know for sure if the qubes-firewall-user-script (located at /rw/config/) from Solene’s guide translating the previously made iptables killswitch, made the VPN unaccessible somehow. I tried with iptables afterwards with the same poor results.

Now, for those who are looking for the required packages for their wireguard VPN’s on minimal templates:

• debian-12-minimal requires qubes-core-agent-networking qubes-core-agent-network-manager wireguard resolvconf iptables

Notes:
Do not add the --no-install-suggests or --no-install-recommends options as this will break things. Optionally you can install xfce4-notifyd for VPN notifications.
The networking packages install openresolv which conflicts with resolvconf so it uninstalls it.

• fedora-38-minimal requires qubes-core-agent-networking qubes-core-agent-network-manager wireguard-tools xfce4-notifyd

Notes: I didn’t test the dependencies thoroughly but it should work better out of the box than debian-12-minimal.

If somebody wants to troubleshoot even further I made sure to list all installed packages ( apt list | grep installed ) from the minimal debian templates that you can find in the two attached documents (provided in .gz archive since the forums don’t seem to allow txt files, use gzip -d filename).
failing-vpn-template-02.txt.gz (10.4 KB)
working-vpn-template-01.txt.gz (6.0 KB)

Do note that I am using kitty as a terminal.
kitty 's dependencies are as follows:
kitty-shell-integration kitty-terminfo libc6 (>= 2.34) libdbus-1-3 (>= 1.9.14) libharfbuzz0b (>= 1.6.0) liblcms2-2 (>= 2.2+git20110628) libpng16-16 (>= 1.6.2-1) libpython3.11 (>= 3.11.0) librsync2 (>= 2.3.1) libssl3 (>= 3.0.0) libwayland-client0 (>= 1.20.0) libx11-6 (>= 2:1.2.99.901) libx11-xcb1 (>= 2:1.8.4) libxkbcommon-x11-0 (>= 0.5.0) libxkbcommon0 (>= 1.0.0) python3 (<< 3.12) python3.11 zlib1g (>= 1:1.1.4)

Having the recommended packages I also have kitty-doc libcanberra0 with the respective dependencies:
libjs-sphinxdoc (>= 5.2)
libasound2 (>= 1.0.16) libc6 (>= 2.33) libltdl7 (>= 2.4.7) libtdb1 (>= 1.2.7+git20101214 libvorbisfile3 (>= 1.1.2) sound-theme-freedesktop

[solene]

@eBread

My setup usually includes a vpn firewall ( AppVM → VPN qube → vpn qube firewall → sys-firewall) to manage the authorized IPs the VPN can reach,

the qube vpn qube firewall here is useless, the filtering for the VPN qube could be done by sys-firewall like any other VM. However, it may make sense to have it between the AppVM and the VPN qube

@eBread

Solene’s guide translating the previously made iptables killswitch, made the VPN unaccessible somehow

The killswitch blocks forwarding on eth0, which should only be used for qubes behind the one where this code is running, so it shouldn’t block the traffic to establish the VPN itself (except if you put rules in vpn qube firewall and sys firewall that could prevent reaching the VPN server)

@eBread

If somebody wants to troubleshoot even further

I don’t understand, do you still have an issue? If so, could you share some logs? Are you using the VPN provider app or simply network manager?

[eBread]

@solene

the qube vpn qube firewall here is useless

Since the script in the vpn itself doesn’t work it is the only way I found that restricts the outgoing traffic thanks to the qvm-firewall rules in the firewall-vpn qube.

@solene

The killswitch blocks forwarding on eth0, which should only be used for qubes behind the one where this code is running, so it shouldn’t block the traffic to establish the VPN itself

That’s what I understood indeed however it seems to not work unlike in 4.1 where I had the same script in a single vpn qube (with iptables)

@solene

(except if you put rules in vpn qube firewall and sys firewall that could prevent reaching the VPN server)

I tried to put the rules and script individually then together for every possibilities, somehow it didn’t work.

@solene

I don’t understand, do you still have an issue? If so, could you share some logs? Are you using the VPN provider app or simply network manager?

Since my current setup is working (although I wish the script would work instead of this ‘dirty’ hack) and I got a bit fed up I didn’t dig further. I put a good amount of information in the case someone sees this and has the same issue as me so, if they wish, they could potentially come to a conclusion.
To answer your last question, I am using the configuration files with network manager.

[solene]

@eBread

Since the script in the vpn itself doesn’t work it is the only way I found that restricts the outgoing traffic thanks to the qvm-firewall rules in the firewall-vpn qube.

That should work without firewall-vpn, it’s adding nothing

qvm-firewall applies the rules on the NetVM of a qube, whether it’s qvm-firewall or sys-firewall, they should yield the exact same result, aka blocking the non VPN traffic as you configured it. If you have a different result depending on the NetVM qube, something is wrong in your setup.

[eBread]

@solene

qvm-firewall or sys-firewall, they should yield the exact same result

Indeed, however blocking the traffic from sys-firewall would mean blocking it for all other networks passing by it hence the addition of the firewall vpn qube.

If I understand correctly, and please to correct me if I am wrong, in a setup with multiple VPN qubes, sys-whonix and the likes, if I apply the rules inside of sys-firewall (the original qube created at the very beginning by Qubes itself) then I would block all the traffic except for the HOSTS indicated by the qvm-firewall command in dom0.

[eBread]

Also, regarding the post itself, I didn’t quite know how to make a brief and understable title. The current one still bugs me. Would “[4.2] Temporary fix and troubleshooting for minimal templates” even work? I feel like it would be too long.

[solene]

@eBread

Indeed, however blocking the traffic from sys-firewall would mean blocking it for all other networks passing by it hence the addition of the firewall vpn qube.

No, the rules are applied per qubes at the NetVM level. You can block all traffic for one qube and all traffic but one host for another, using the same NetVM and it will work

[eBread]

@solene

You can block all traffic for one qube and all traffic but one host for another, using the same NetVM

I must admit I am confused here. We may or may not be talking about the same thing or procedure but I seem to have lost track of it.

I am attaching what I am personally refering to as an image. The rules I am talking about are applied from dom0 with the command qvm-firewall. However I have lost track if you were talking about the nftables and/or iptables rules inside the NetVM or outside.

On that note I will come by at a later date. Thank you, I appreciate the input. Have a nice day or night.

[DVM]

What @solene was trying to say is that what you see here on your screenshot is not applied inside the qube itself, but inside its netvm. If you open a terminal in the netvm and list the qubes-firewall table (sudo nft list table ip qubes-firewall), you will see that all your rules are there. If you really want to use the “sys-vpn-fw-01” qube correctly, you need to move these rules to the qube(s) that use it as a netvm so that the rules are applied in the fw qube.

[ddungtang]

I’ve too been troubleshooting issues with my VPN qube which is also based on a minimal template (although fedora minimal, instead of debian). At first, I suspected that wireguard VPN broke due to the template upgrade from 4.1 to 4.2, either to leaving junk behind, or missing some new required packages. I’ve recreated the template, and wireguard VPN worked. However, as soon as I added MAC randomization (Anonymizing your MAC address) the VPN qube stopped being able to connect to the VPN server.

[eBread]

Interesting, thank you for the clarification and thanks to @solene for her previous explanations. I wasn’t aware of this actually, and it does makes sense with some of what I’ve been seeing with the qhole ‘qvm-firewall’ command that I was not able to grasp beforehand.

@ddungtang

I’ve recreated the template, and wireguard VPN worked. However, as soon as I added MAC randomization (Anonymizing your MAC address ) the VPN qube stopped being able to connect to the VPN server.

I see, that is good to know, thanks. I haven’t had the time to make my custom sys-net qube on 4.2, this will definitely save me a lot of hassle.

[PostedPortal]

in another thread it was explained to me that 4.2 dropped ip tables support for nftables. Its beyond my knowledge why but specifically mac address randomization will break sys-vpn on 4.2.

As long as mac address randomization is on for sys-net qubes you are fine. Because the ISP will not be able to see the mac address of the sys vpns beyond sys-net. Applications on sys-vpn or other APPvms will read their mac address as the same default setting for all app vms ( from what i was told)