Network no longer work after upgrade to 4.2

Hi, I just completed upgrade to Qubes 4.2

I’m having network issues: network work ONLY in sys-net, any other qube has no internet access (maybe a DNS issue?).

I tried to manually run sudo /usr/lib/qubes/qubes-setup-dnat-to-ns in sys-net but the issue persist.

This issue started in STAGE 4, initially probably caused by mirage firewall, that I removed and replaced with sys-firewall (solving the libxenlight issues but not the network issues).

Anyone had the same problem and know how to fix it?

Can you ping IP addresses from sys-firewall?
Try to ping in sys-firewall for a test:

ping 9.9.9.9
ping quad9.com

No, I tried but not work. Ping from sys-net work. From any other qube (including sys-firewall) not work.

Can you post the output of this command in sys-net and sys-firewall?

sudo nft list ruleset
user@sys-net:~$ sudo nft list ruleset
table ip qubes {
	set downstream {
		type ipv4_addr
	}

	set allowed {
		type ifname . ipv4_addr
	}

	chain prerouting {
		type filter hook prerouting priority raw; policy accept;
		iifgroup 2 goto antispoof
		ip saddr @downstream counter packets 0 bytes 0 drop
	}

	chain antispoof {
		iifname . ip saddr @allowed accept
		counter packets 0 bytes 0 drop
	}

	chain postrouting {
		type nat hook postrouting priority srcnat; policy accept;
		oifgroup 2 accept
		oif "lo" accept
		masquerade
	}

	chain input {
		type filter hook input priority filter; policy drop;
		jump custom-input
		ct state invalid counter packets 0 bytes 0 drop
		iifgroup 2 udp dport 68 counter packets 0 bytes 0 drop
		ct state established,related accept
		iifgroup 2 meta l4proto icmp accept
		iif "lo" accept
		iifgroup 2 counter packets 0 bytes 0 reject with icmp host-prohibited
		counter packets 10 bytes 2316
	}

	chain forward {
		type filter hook forward priority filter; policy accept;
		jump custom-forward
		ct state invalid counter packets 0 bytes 0 drop
		ct state established,related accept
		oifgroup 2 counter packets 0 bytes 0 drop
	}

	chain custom-input {
	}

	chain custom-forward {
	}

	chain dnat-dns {
		type nat hook prerouting priority dstnat; policy accept;
		ip daddr 10.139.1.1 udp dport 53 dnat to 192.168.2.1
		ip daddr 10.139.1.1 tcp dport 53 dnat to 192.168.2.1
	}
}
table ip6 qubes {
	set downstream {
		type ipv6_addr
	}

	set allowed {
		type ifname . ipv6_addr
	}

	chain antispoof {
		iifname . ip6 saddr @allowed accept
		counter packets 0 bytes 0 drop
	}

	chain prerouting {
		type filter hook prerouting priority raw; policy accept;
		iifgroup 2 goto antispoof
		ip6 saddr @downstream counter packets 0 bytes 0 drop
	}

	chain postrouting {
		type nat hook postrouting priority srcnat; policy accept;
		oifgroup 2 accept
		oif "lo" accept
		masquerade
	}

	chain _icmpv6 {
		meta l4proto != ipv6-icmp counter packets 0 bytes 0 reject with icmpv6 admin-prohibited
		icmpv6 type { nd-router-advert, nd-redirect } counter packets 0 bytes 0 drop
		accept
	}

	chain input {
		type filter hook input priority filter; policy drop;
		jump custom-input
		ct state invalid counter packets 0 bytes 0 drop
		ct state established,related accept
		iifgroup 2 goto _icmpv6
		iif "lo" accept
		ip6 saddr fe80::/64 ip6 daddr fe80::/64 udp dport 546 accept
		meta l4proto ipv6-icmp accept
		counter packets 0 bytes 0
	}

	chain forward {
		type filter hook forward priority filter; policy accept;
		jump custom-forward
		ct state invalid counter packets 0 bytes 0 drop
		ct state established,related accept
		oifgroup 2 counter packets 0 bytes 0 drop
	}

	chain custom-input {
	}

	chain custom-forward {
	}
}
table ip qubes-firewall {
	chain forward {
		type filter hook forward priority filter; policy drop;
		ct state established,related accept
		iifname != "vif*" accept
		ip saddr 10.137.0.32 jump qbs-10-137-0-32
	}

	chain prerouting {
		type filter hook prerouting priority raw; policy accept;
		iifname != "vif*" ip saddr 10.137.0.32 drop
	}

	chain postrouting {
		type filter hook postrouting priority raw; policy accept;
		oifname != "vif*" ip daddr 10.137.0.32 drop
	}

	chain qbs-10-137-0-32 {
		accept
		reject with icmp admin-prohibited
	}
}
table ip6 qubes-firewall {
	chain forward {
		type filter hook forward priority filter; policy drop;
		ct state established,related accept
		iifname != "vif*" accept
	}

	chain prerouting {
		type filter hook prerouting priority raw; policy accept;
	}

	chain postrouting {
		type filter hook postrouting priority raw; policy accept;
	}
}
user@sys-firewall:~$ sudo nft list ruleset
table ip qubes {
	set downstream {
		type ipv4_addr
	}

	set allowed {
		type ifname . ipv4_addr
	}

	chain prerouting {
		type filter hook prerouting priority raw; policy accept;
		iifgroup 2 goto antispoof
		ip saddr @downstream counter packets 0 bytes 0 drop
	}

	chain antispoof {
		iifname . ip saddr @allowed accept
		counter packets 0 bytes 0 drop
	}

	chain postrouting {
		type nat hook postrouting priority srcnat; policy accept;
		oifgroup 2 accept
		oif "lo" accept
		masquerade
	}

	chain input {
		type filter hook input priority filter; policy drop;
		jump custom-input
		ct state invalid counter packets 0 bytes 0 drop
		iifgroup 2 udp dport 68 counter packets 0 bytes 0 drop
		ct state established,related accept
		iifgroup 2 meta l4proto icmp accept
		iif "lo" accept
		iifgroup 2 counter packets 0 bytes 0 reject with icmp host-prohibited
		counter packets 0 bytes 0
	}

	chain forward {
		type filter hook forward priority filter; policy accept;
		jump custom-forward
		ct state invalid counter packets 0 bytes 0 drop
		ct state established,related accept
		oifgroup 2 counter packets 0 bytes 0 drop
	}

	chain custom-input {
	}

	chain custom-forward {
	}

	chain dnat-dns {
		type nat hook prerouting priority dstnat; policy accept;
		ip daddr 10.139.1.1 udp dport 53 dnat to 10.139.1.1
		ip daddr 10.139.1.1 tcp dport 53 dnat to 10.139.1.1
		ip daddr 10.139.1.2 udp dport 53 dnat to 10.139.1.2
		ip daddr 10.139.1.2 tcp dport 53 dnat to 10.139.1.2
	}
}
table ip6 qubes {
	set downstream {
		type ipv6_addr
	}

	set allowed {
		type ifname . ipv6_addr
	}

	chain antispoof {
		iifname . ip6 saddr @allowed accept
		counter packets 0 bytes 0 drop
	}

	chain prerouting {
		type filter hook prerouting priority raw; policy accept;
		iifgroup 2 goto antispoof
		ip6 saddr @downstream counter packets 0 bytes 0 drop
	}

	chain postrouting {
		type nat hook postrouting priority srcnat; policy accept;
		oifgroup 2 accept
		oif "lo" accept
		masquerade
	}

	chain _icmpv6 {
		meta l4proto != ipv6-icmp counter packets 0 bytes 0 reject with icmpv6 admin-prohibited
		icmpv6 type { nd-router-advert, nd-redirect } counter packets 0 bytes 0 drop
		accept
	}

	chain input {
		type filter hook input priority filter; policy drop;
		jump custom-input
		ct state invalid counter packets 0 bytes 0 drop
		ct state established,related accept
		iifgroup 2 goto _icmpv6
		iif "lo" accept
		ip6 saddr fe80::/64 ip6 daddr fe80::/64 udp dport 546 accept
		meta l4proto ipv6-icmp accept
		counter packets 0 bytes 0
	}

	chain forward {
		type filter hook forward priority filter; policy accept;
		jump custom-forward
		ct state invalid counter packets 0 bytes 0 drop
		ct state established,related accept
		oifgroup 2 counter packets 0 bytes 0 drop
	}

	chain custom-input {
	}

	chain custom-forward {
	}
}
table ip qubes-firewall {
	chain forward {
		type filter hook forward priority filter; policy drop;
		ct state established,related accept
		iifname != "vif*" accept
		ip saddr 10.137.0.38 jump qbs-10-137-0-38
		ip saddr 10.137.0.7 jump qbs-10-137-0-7
		ip saddr 10.137.0.23 jump qbs-10-137-0-23
		ip saddr 10.137.0.31 jump qbs-10-137-0-31
		ip saddr 10.137.0.21 jump qbs-10-137-0-21
	}

	chain prerouting {
		type filter hook prerouting priority raw; policy accept;
		iifname != "vif*" ip saddr { 10.137.0.7, 10.137.0.21, 10.137.0.23, 10.137.0.31, 10.137.0.38 } drop
	}

	chain postrouting {
		type filter hook postrouting priority raw; policy accept;
		oifname != "vif*" ip daddr { 10.137.0.7, 10.137.0.21, 10.137.0.23, 10.137.0.31, 10.137.0.38 } drop
	}

	chain qbs-10-137-0-38 {
		accept
		reject with icmp admin-prohibited
	}

	chain qbs-10-137-0-7 {
		accept
		reject with icmp admin-prohibited
	}

	chain qbs-10-137-0-23 {
		accept
		reject with icmp admin-prohibited
	}

	chain qbs-10-137-0-31 {
		accept
		reject with icmp admin-prohibited
	}

	chain qbs-10-137-0-21 {
		accept
		reject with icmp admin-prohibited
	}
}
table ip6 qubes-firewall {
	chain forward {
		type filter hook forward priority filter; policy drop;
		ct state established,related accept
		iifname != "vif*" accept
	}

	chain prerouting {
		type filter hook prerouting priority raw; policy accept;
	}

	chain postrouting {
		type filter hook postrouting priority raw; policy accept;
	}
}

Did you run sudo nft list ruleset in sys-net when sys-fiwarall was running and connected to sys-net?
For some reason the sys-firewall IP is not present in downstream and allow sets, it should’ve been like this:

table ip qubes {
	set downstream {
		type ipv4_addr
		elements = { 10.137.0.32 }
	}

	set allowed {
		type ifname . ipv4_addr
		elements = { "vifX.0" . 10.137.0.32 }
	}
```
But at the same time sys-fireweall IP is present in `qubes-firewall` forward chain.
Not sure what is broken.
Do you use template updated from Qubes OS 4.1 to Qubes OS 4.2 for sys-net?
Maybe the upgrade failed somewhere.
I'd suggest to install clean Qubes OS 4.2 templates and use them.

Thanks for reply

I fixed by installing fresh debain-12-minimal template for network VMs (with network-related packages).

Now it’s fixed, would you mind to try out again with mirage-firewall? I’m interested to fix any potential issue here. I’m also on up to date 4.2 and I don’t have issues (even with 32Mb memory).