Networking broken in Debian-based templates after R4.2 upgrade

User Support General
,
1

I updated to R4.2 yesterday, so I believe this is related.

Setup

Internet ↔ sys-net ↔ sys-net-fw ↔ vpn-qube ↔ vpn-qube-fw ↔ various qubes

  • The vpn-qube has a firewall setting to restrict traffic to only the VPN endpoint.
  • I have a wrapper script around my VPN client that runs the qubes-setup-dnat-to-ns.sh script after connecting to the VPN.
  • The VPN client creates a tun0 interface that should have all traffic routed through it (i.e., a full tunnel).

This was functional before the upgrade to 4.2.

Functioning

  1. ICMP works from vpn-qube.
  2. VPN connects from vpn-qube.
  3. dnat-dns entries are being made in vpn-qube

Broken

Nothing works upstream of the vpn-qube.

  1. ICMP from vpn-qube-fw and all connected qubes.
  2. DNS is broken even when I specify the resolving server (e.g., from vpn-qube-fw dig @vpn_dns_server google.com)

Edit to add

When I attempt to change the network configuration for vpn-qube-fw, the Qubes Manager GUI crashes. Attempting to change from the shell fails with the following:

$ qvm-prefs vpn-qube-fw netvm sys-net
...
qvm-prefs: error: no such property: 'netvm'

However:

$ qvm-prefs vpn-qube-fw | grep netvm
netvm                    -   vpn-qube
2

Attempting to change from the shell fails with the following:

There used to be an issue with this when the old netvm is not running, but I think it should be fixed. Have you tried this though?

3

Powering off vpn-qube-fw resolved the netvm changing problem. Both sys-net and vpn-qube were powered on at the time. After powering off vpn-qube-fw, I was able to change its netvm to sys-net.

With sys-net, vpn-qube-fw was able to ping and reach Internet hosts (e.g., ping 8.8.8.8).

Reviewing similar issues, I think the problem is in the vpn-qube not updating its downstream and allowed configuration. From https://forum.qubes-os.org/t/network-no-longer-work-after-upgrade-to-4-2/23081/6, I also do not see any entries for vpn-qube-fw in vpn-qube; however, when I switched vpn-qube-fw to sys-net, I saw the vpn-qube-fw IP address show up as an entry (in sys-net).

I have powered off both vpn-qube and vpn-qube-fw, and re-set the netvm to vpn-qube with no change to the nft ruleset in vpn-qube.

4

The issue appears to be related to Debian-based templates.

  • debian-12-minimal [original] → broken
  • debian-12 → broken
  • fedora-38 → functional

[Debian 12 vpn-qube] Powering on vpn-qube-fw shows:

vif vif-44-0 vif44.0: Guest Rx ready

But, nothing else appears to happen.

[Fedora 38] Powering on vpn-qube-fw shows a bunch of text and updates being made to nft.

5

Did packages change for Debian 12 networking in R4.2? Networking is completely broken for me in Debian-based templates. I swapped my sys-net to debian-12 and I can replicate the broken behavior. Switching to Fedora 38 as a template restores networking.

This is all independent and before my VPN client is even touched, to be clear.

6

To be clear, networking works for the Debian-based network qube itself, but there is no network connectivity for devices that use it as a netvm.

7

Did you only try the debian templates upgraded from 4.1 to 4.2?
Try to install an use clean Qubes OS 4.2 debian template for a test.

8

Did you do an in-place upgrade or a clean install with backup restoration?
Can you check in /etc/apt/sources.list.d/qubes-r4.list if you have r4.2 for each repos?

9

Let’s assume that a clean Debian install would resolve the issue, but is not an option at the moment.

In-place upgrade.

Checking /etc/apt/sources.list.d/:

/etc/apt/sources.list.d# ls -l
total 20
-rw-r--r-- 1 root root  375 Sep 28 09:30 qubes-contrib-r4.1.list
-rw-r--r-- 1 root root  360 Jul 23 02:09 qubes-contrib-r4.2.list
-rw-r--r-- 1 user user 2835 Dec 23 19:21 qubes-r4.list
-rw-r--r-- 1 root root 2355 Dec 23 19:21 qubes-r4.list.bak
-rw-r--r-- 1 user user 2955 Nov 13 20:32 qubes-r4.list.dpkg-dist

Just at first glance that is not a good sign. The qubes-contrib-r4.1.list is still present and the ownership of qubes-r4.list and qubes-r4.list.dpkg-dist are user.

qubes-r4.list does not contain any r4.1 references. It appears to have been successfully modified for r4.2.

# grep r4.1 qubes-r4.list
# grep r4.2 qubes-r4.list
deb [arch=amd64 signed-by=/usr/share/keyrings/qubes-archive-keyring-4.2.gpg signed-by=/usr/share/keyrings/qubes-archive-keyring-4.2.gpg] https://deb.qubes-os.org/r4.2/vm bookworm main
[snip]

I’ve commented out the r4.1 deb source in qubes-contrib-r4.1.list and also changed the other source files’ ownership to root.

Running apt dist-upgrade only mentioned some packages for removal, so I let that process. I’m going to see if this changed/fixed anything and see if I can find what else qubes-dist-upgrade -t may have been missed.

Also, for clarity, the above is from a modified debian-minimal template but as mentioned earlier, the symptoms were present when I switched to my debian-12 template. My debian-12 template should be a stock template.

/etc/apt$ ls -la sources.list.d/
total 20
drwxr-xr-x 2 root root 4096 Dec 23 19:22 .
drwxr-xr-x 8 root root 4096 Jul 23 20:42 ..
-rw-r--r-- 1 root root 2835 Dec 23 19:22 qubes-r4.list
-rw-r--r-- 1 root root 2355 Dec 23 19:22 qubes-r4.list.bak
-rw-r--r-- 1 root root 2955 Nov 13 20:32 qubes-r4.list.dpkg-dist
/etc/apt$ grep r4.2 sources.list.d/qubes-r4.list
deb [arch=amd64 signed-by=/usr/share/keyrings/qubes-archive-keyring-4.2.gpg signed-by=/usr/share/keyrings/qubes-archive-keyring-4.2.gpg] https://deb.qubes-os.org/r4.2/vm bookworm main
[snip]
10

The answer to this was found and posted to Github:

I can confirm it works.