Qubes-tunnel blocking all traffic (including vpn)?

Alethia · February 9, 2022, 9:43pm

I just installed the proper Qubes 4.1 version (been running the alpha 4.1 a long time) and now I can’t seem to get the qubes-tunnel to work. If I enable the qubes-tunnel service in my vpn qube then all outgoing traffic seems to be blocked (including the VPN traffic) resulting in ping not working as well as the VPN failing its DNS queries:

systemd[1]: Starting Tunnel service for Qubes proxyVM...
su[1050]: (to user) root on none
su[1050]: pam_unix(su-l:session): session opened for user user(uid=1000) by (uid=0)
su[1050]: pam_unix(su-l:session): session closed for user user
qtunnel-setup[1069]: START-ing network forwarding!
qtunnel-setup[1068]: EXEC /usr/sbin/openvpn --cd /rw/config/qtunnel/ --config /tmp/qtunnel.conf --verb 3 --mlock --ping 10 --ping-restart 42 --connect-retry 5 30 --connect-retry-max 7 --resolv-retry 15 --group qtunnel --script-security 2 --up "/usr/lib/qubes/qtunnel-connect up" --down "/usr/lib/qubes/qtunnel-connect down" --auth-user-pass /tmp/tunneluserpwd.txt
systemd[1]: Started Tunnel service for Qubes proxyVM.
qtunnel-setup[1072]: 2022-02-09 22:28:15 DEPRECATED OPTION: --cipher set to 'AES-256-CBC' but missing in --data-ciphers (AES-256-GCM:AES-128-GCM). Future OpenVPN version will ignore --cipher for cipher negotiations. Add 'AES-256-CBC' to --data-ciphers or change --cipher 'AES-256-CBC' to --data-ciphers-fallback 'AES-256-CBC' to silence this warning.
qtunnel-setup[1072]: 2022-02-09 22:28:15 OpenVPN 2.5.5 x86_64-redhat-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Jan 27 2022
qtunnel-setup[1072]: 2022-02-09 22:28:15 library versions: OpenSSL 1.1.1l  FIPS 24 Aug 2021, LZO 2.10
qtunnel-setup[1072]: 2022-02-09 22:28:15 mlockall call succeeded
qtunnel-setup[1072]: 2022-02-09 22:28:15 WARNING: you are using user/group/chroot/setcon without persist-tun -- this may cause restarts to fail
qtunnel-setup[1072]: 2022-02-09 22:28:15 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
qtunnel-setup[1072]: 2022-02-09 22:28:15 Outgoing Control Channel Authentication: Using 512 bit message hash 'SHA512' for HMAC authentication
qtunnel-setup[1072]: 2022-02-09 22:28:15 Incoming Control Channel Authentication: Using 512 bit message hash 'SHA512' for HMAC authentication
qtunnel-setup[1072]: 2022-02-09 22:28:15 RESOLVE: Cannot resolve host address: se.ovpn.azirevpn.net:1194 (Temporary failure in name resolution)
qtunnel-setup[1072]: 2022-02-09 22:28:15 RESOLVE: Cannot resolve host address: se.ovpn.azirevpn.net:1194 (Temporary failure in name resolution)
qtunnel-setup[1072]: 2022-02-09 22:28:20 RESOLVE: Cannot resolve host address: se.ovpn.azirevpn.net:1194 (Temporary failure in name resolution)
qtunnel-setup[1072]: 2022-02-09 22:28:25 RESOLVE: Cannot resolve host address: se.ovpn.azirevpn.net:1194 (Temporary failure in name resolution)
qtunnel-setup[1072]: 2022-02-09 22:28:25 Could not determine IPv4/IPv6 protocol
qtunnel-setup[1072]: 2022-02-09 22:28:25 NOTE: UID/GID downgrade will be delayed because of --client, --pull, or --up-delay
qtunnel-setup[1072]: 2022-02-09 22:28:25 SIGUSR1[soft,init_instance] received, process restarting
qtunnel-setup[1072]: 2022-02-09 22:28:25 Restart pause, 5 second(s)

This is the output from sudo journalctl -u qubes-tunnel. I’m also receiving the continuous popup notifications stating ‘Ready to start link’.

Without the qubes-tunnel service I can ping and connect to the internet just fine, but obviously there’s no VPN.

This VPN configuration works fine on my other devices currently.

Am I missing something here?

Alethia · March 16, 2022, 9:56am

Managed to solve this by reverting from Fedora 34 to 32.

@fepitre do you know of any issues related to Fedora 34?

deeplow · March 16, 2022, 10:04am

Alethia:

qtunnel-setup[1072]: 2022-02-09 22:28:15 RESOLVE: Cannot resolve host address: se.ovpn.azirevpn.net:1194 (Temporary failure in name resolution)
qtunnel-setup[1072]: 2022-02-09 22:28:15 RESOLVE: Cannot resolve host address: se.ovpn.azirevpn.net:1194 (Temporary failure in name resolution)
qtunnel-setup[1072]: 2022-02-09 22:28:20 RESOLVE: Cannot resolve host address: se.ovpn.azirevpn.net:1194 (Temporary failure in name resolution)
qtunnel-setup[1072]: 2022-02-09 22:28:25 RESOLVE: Cannot resolve host address: se.ovpn.azirevpn.net:1194 (Temporary failure in name resolution)

Seems to be an internet connection issue.

Alethia · March 16, 2022, 10:44am

Yes, question is why there would be a connection issue with Fedora 34 and not with Fedora 32

My other VMs running Fedora 34 can access se.ovpn.azirevpn.net just fine…

fepitre · March 17, 2022, 2:35pm

Not really, I would recommend you to use bullseye. I’m using it especially due to recent random issues I had with VPN in the last years when bumping between Fedora versions like you.

tetrahedra · April 30, 2022, 7:33pm

Unfortunately Debian-11 (Bullseye) doesn’t work either.

Under Debian, the VPN appears to connect. However, no traffic is routed from the AppVM out through the VM (applications in the AppVM don’t work). The VPN then disconnects with the following error:

Apr 30 19:21:47 my-VPN-2 qtunnel-setup[2645]: 2022-04-30 19:21:47 SIGUSR1[soft,ping-restart] received, process restarting
Apr 30 19:21:47 my-VPN-2 qtunnel-setup[2645]: 2022-04-30 19:21:47 Restart pause, 5 second(s)

Using tcpdump -i eth0, I can see that while the VPN is nominally up, traffic flows from the remote endpoint to the VPN VM, but the VPN VM does not reply.

However, traffic flows in both directions during the process of establishing the initial VPN connection.

Using tcpdump -i tun0 I can see that traffic is going from the local IP (10.5.0.4) of the VPN VM to the remote IP (10.5.0.1) but no replies are coming back.

Using tcpdump -i vif48.0 I see traffic coming from the AppVM to the VPN VM, but not receiving any reply.

Here is the output of route (on the VPN VM) when the VPN is down:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         10.137.0.21     0.0.0.0         UG    0      0        0 eth0
10.137.0.21     0.0.0.0         255.255.255.255 UH    0      0        0 eth0
10.138.28.51    0.0.0.0         255.255.255.255 UH    32704  0        0 vif48.0

Here is the output of route on the VPN VM when the VPN is nominally up:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.5.0.1        128.0.0.0       UG    0      0        0 tun0
default         10.137.0.21     0.0.0.0         UG    0      0        0 eth0
10.5.0.0        0.0.0.0         255.255.0.0     U     0      0        0 tun0
10.137.0.21     0.0.0.0         255.255.255.255 UH    0      0        0 eth0
10.138.28.51    0.0.0.0         255.255.255.255 UH    32704  0        0 vif48.0
128.0.0.0       10.5.0.1        128.0.0.0       UG    0      0        0 tun0
185.213.x.x 10.137.0.21     255.255.255.255 UGH   0      0        0 eth0

(where 185.213.x.x is the IP of the foreign VPN endpoint)

This also applies to Qubes-vpn-support.

user111 · May 14, 2022, 2:30am

I’ve been using Rudd-O’s VPN on Qubes, and like Tasket’s, it worked until 34. There’s some common element that hasn’t been shared with the wider community yet.

tetrahedra · May 15, 2022, 5:01pm

I tested Qubes-vpn-support (the old version of Qubes-tunnel) on a fedora-33 template on Qubes 4.1.

This combination (which definitely worked fine on Qubes 4.0) does not work on my install of 4.1. The same restart loop behaviour is visible, with the VPN connection timing out due to “inactivity”.

The problem likely lies in the firewall rules and/or routes created after the VPN connects. The VPN is able to connect initially (both with debian-11/Qubes-tunnel and fedora-33/Qubes-vpn-handler) but no traffic flows after that.

An earlier test (likely Debian-11/Qubes-tunnel) showed that even with # sudo --group=qtunnel w3m google.com no traffic was being allowed out from the group which is supposed to be able to send traffic.

tetrahedra · June 2, 2022, 7:37pm

Found a temporary workaround:

The problem (as described in the link above) is that Qubes 4.1 changed how it handles ARP.

tzwcfq · June 2, 2022, 7:48pm

github.com/QubesOS/qubes-issues

Qubes 4.1 - VPN over Tor netvms: ARP request does not get resolved properly

opened 03:03PM - 15 Dec 21 UTC

unknown-ter

T: bug C: core P: default diagnosed C: networking

### Qubes OS release Qubes 4.1 RC2 ### Brief summary A VPN Gateway …over Tor worked fine with Qubes 4.0. It does not work with Qubes 4.1 anymore. ARP requests aren't proxied properly. ### Steps to reproduce Configure VPN gateway as described [in the Contrib docs](https://github.com/Qubes-Community/Contents/blob/master/docs/configuration/vpn.md). Configure the Tor gateway according to the steps described [in Whonix docs](https://www.whonix.org/wiki/Tunnels/Connecting_to_Tor_before_a_VPN#Separate_VPN-Gateway). Then: ``` ping 1.1.1.1 # or 8.8.8.8 ``` Qubes 4.0 -> OK Qubes 4.1 -> No resolution ### Expected behavior Same behavior as Qubes 4.0 ### Actual behavior ARP requests are not getting proxied properly / do not receive the MAC address fe:ff:ff:ff:ff:ff of the corresponding netVM. [Sample](https://forum.qubes-os.org/t/user-tor-vpn-internet-4-1-broken/6900) from user `aUsername`: ``` bash-5.1# ip neigh show 10.137.0.6 dev eth0 lladdr fe:ff:ff:ff:ff:ff PERMANENT 212.129.0.80 dev eth0 INCOMPLETE ``` , whereas 212.129.0.80 in this case is the <IP> of the remote VPN gateway. `aUsename` also mentions a fix: ``` arp -s <IP> -i eth0 fe:ff:ff:ff:ff:ff ``` This manual correction has not been necessary with Qubes 4.0. The Whonix maintainer states [here](http://forums.dds6qkxpwdeubwucdiaord2xgbbeyds25rbsgr73tbfpqpt4a6vjwsyd.onion/t/qubes-4-1-tor-openvpn-issues/13006/4), there haven't been related changes in the Whonix configuration: > You could report this issue on qubes-issues because nothing related in the Whonix configuration changed , no such issue with Non-Qubes-Whonix reported. Therefore some change between Qubes 4.0 and Qubes R4.1 might have caused this. ### Related issues - https://forum.qubes-os.org/t/user-tor-vpn-internet-4-1-broken - https://forum.qubes-os.org/t/networking-broken-in-4-1-default-templates - https://forum.qubes-os.org/t/tor-vpn-connection-issues-only-in-4-1-multiple-test-configurations-vpn-providers - https://forum.qubes-os.org/t/r4-1-rc1-openvpn-issue-after-migration-from-r4-0