New and older installs have no networking outside sys-net

User Support
1

Thank you so much for reading my post.

I am rather at my wit’s end with this issue, which totally cripples using Qubes. It must be the result of a critical bug.

Here’s my story:

  1. I bought a 2013 Dell laptop (not on HCL) a year ago and installed Qubes. After replacing the network card, I was able to get online with sys-net, but no other VMs had any connectivity. I did a lot of searches but never found an answer on how to fix this issue – until one day it magically went away and I was able to set up my Qubes system, which went swimmingly. Great OS! (or so I thought…)
  2. Fast forward to a few weeks ago. While I was away from the computer at a random point in time, my online processes in VMs simply stopped being able to get online. The bug had returned! I spent many hours researching, but my networking knowledge is very limited so I wasn’t able to get very far. I hadn’t changed any settings, although I had been keeping up with Qubes updates.
  3. I tried numerous things to solve the issue: changing templates of sys-net, sys-firewall etc. to Fedora 29,30,31,32… making a new sys-net from scratch, restoring a backup from when it still worked, installing 4.0.3 and 4.0.4rc from scratch and more, all to no avail. sys-net happily connected to wifi, but no other qubes could get any connectivity from it.
  4. I found some posts online that suggested the problem may be a race condition due to slow hardware, so I bought a new computer that IS on the HCL and installed 4.0.3 fresh on it. SAME ISSUE AGAIN.
  5. I just tried installing 4.0.1 fresh on this new computer and still have ZERO CONNECTIVITY IN VMs outside of sys-net. I can’t do updates or anything.

I am all out of ideas now, having dropped a lot of money on a new computer I didn’t need and now seeing that all my fresh installs of Qubes are totally broken. Are others having this same issue? If so, that will certainly kill usage of this OS. It seems like an absolutely crippling critical bug to leave open.

I have found a number of very similar issues to mine on Github, one of which was actually closed (5570):

/5576
/5570
/5291
/5269

Of course I have tried all the straightforward fixes provided in those issue discussions, but nothing solved the issue. It would make sense that my issue is a race condition, considering how it both disappeared and reappeared out of nowhere. To be clear, I don’t actually know for sure if any of those issues is the same as mine.

To reiterate, sys-net connects to WiFi perfectly, but no other qubes are online. I try running Firefox and/or pinging 8.8.8.8 in sys-firewall for example, and get no response.

Running sudo iptables -t nat -L in sys-net gives me for the Chain PR-QBS 4 DNAT entries, 1 UDP and 1 TCP for each 10.139.1.1 & 10.139.1.2. I found that command on one of the issues above; my knowledge of networking is very basic so I’m not really sure what I’m looking at.

I tried running all the suggestions in 5648 above – no change.

Questions

  1. Can you recommend a fix for my issue?
  2. If you think Issue 5570 is the same as mine, I’d like to try to download the fix, but of course all my VMs are offline but sys-net. Can you recommend a workaround so my VMs can get online for an update, somehow? (I tried adding networking features to my default Fedora 29 template but couldn’t figure out how to get online as no connection could be added via the GUI).

Be easy on me, please. I have rudimentary command line abilities and only basic networking knowledge.

Thank you very kindly
—A frustrated Qubes user

1 Like
2

Very frustrating indeed.

Are you currently using 4.1 or 4.0?
It would help if you could say what network card you are using.
What template are you using for sys-net? Have you tried switching
template?

In sys-net, with sys-firewall connected, and after attempting connection
from sys-firewall, what is the result of:
ip route
nft list table qubes-firewall
iptables -t raw -L -nv
iptables -t nat -L -nv

On sys-firewall, what is the result of:
ip route

I very much doubt that #5570 is yours, and in any case you should
already have the fix for that installed.

As to how to get patches etc when sys-net is not working, the simplest
route is to use a live distribution written to a USB stick. Fedora,
Debian, Knoppix: there are many live distributions available.
I always recommend that users have one of these in the drawer just in
case.
You can boot from them, download any drivers or patches, and store them on the
stick, (if you have partitioned it with spare space). Then when you boot
Qubes you can access the files on the stick.
This does, of course, require another computer that is online to get the
live distro.

Finally, get rid of that 29 template 9as quickly as you can- it’s
woefully out of date and not supported.

2 Likes
3

Thank you for your reply, unman.

As mentioned in the OP, I have tried 4.0.3, 4.0.4rc, and 4.0.1 and am seeing this error on all fresh installs of those, as well as my year-old install of 4.0.3 with all updates. It sounds like 4.1 is still pretty unstable so I haven’t tried it yet as I need stability in my setup.

I don’t see why the network card should matter, since I’m connecting to WiFi perfectly inside sys-net. Regardless, the two computers I’m testing that exhibit this “no internet connection in VMs” bug have two different network cards. One of them is an Intel Wireless 8260 and the other I think is also Intel, but that machine is completely out of order right now so I really can’t remember. I’ve never had an issue with either WiFi card.

Yes, I tried switching templates as also mentioned in OP. I tried out all the Fedora (29,30,31,32,33) templates under sys-net and sys-firewall with no difference in this bug. I also tried Debian but could not get sys-net to connect to wifi. Not sure how that is done.

I know that Fedora 29 and Deb 9 are outdated, but I was desperate and saw that one of the Github issues had comments to the effect that the bug started in 4.0.2, so I tried installing 4.0.1 fresh just as a test (with no success). To clarify, the Qubes install that started to get this bug all of a sudden (again) was 4.0.3 WITH all the updates I could install up to a few days before the bug resurfaced, after which point I was unable to update any VMs but sys-net easily.

Here are the results of the commands in sys-net you asked for! (Copying by hand here so hopefully no typos)

$ ip route
default via 172.16.0.1 dev wls7 proto dhcp metric 600
10.137.0.6 dev vif31.0 scope link metric 32721
172.16.0.0/21 dev wls7 proto kernel scope link src 172.16.5.4 metric 600

$ nft list table qubes-firewall
Error: Could not process rule: No such file or directory
list table qubes firewall
[underline under qubes firewall]

$ sudo iptables -t raw -L -nv
Chain PREROUTING (policy ACCEPT 24009 packets, 103M bytes)
pkts bytes target prot opt in out source destination
0 0 DROP all -- vif31.0 * !10.137.0.6 0.0.0.0/0
0 0 DROP all -- vif32.0 * !10.137.0.6 0.0.0.0/0

Chain OUTPUT (policy ACCEPT 16606 packets, 1257K bytes)
pkts bytes target prot opt in out source destination

I guess there was no output data on that one.

$ sudo iptables -t nat -L -nv
Chain PREROUTING (policy ACCEPT 1597 packets, 74492 bytes)
pkts bytes target prot opt in out source destination
2324 128K PR-QBS all -- * * 0.0.0.0/0 0.0.0.0/0
1599 75150 PR-QBS-SERVICES all -- * * 0.0.0.0/0 0.0.0.0/0
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination

Chain OUTPUT (policy ACCEPT 158 packets, 10450 bytes)
pkts bytes target prot opt in out source destination

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- * vif+ 0.0.0.0/0 0.0.0.0/0
1 52 ACCEPT all -- * lo 0.0.0.0/0 0.0.0.0/0
950 66789 MASQUERADE all -- * * 0.0.0.0/0 0.0.0.0/0

Chain PR-QBS (1 references)
pkts bytes target prot opt in out source destination
363 26314 DNAT udp -- * * 0.0.0.0/0 10.139.1.1 udp dpt:53 to:192.168.84.120
0 0 DNAT tcp -- * * 0.0.0.0/0 10.139.1.1 tcp dpt:53 to:192.168.84.120
362 26245 DNAT udp -- * * 0.0.0.0/0 10.139.1.2 udp dpt:53 to:192.168.84.120
0 0 DNAT tcp -- * * 0.0.0.0/0 10.139.1.2 tcp dpt:53 to:192.168.84.120

Chain PR-QBS-SERVICES (1 references)
pkts bytes target prot opt in out source destination
0 0 REDIRECT tcp -- vif+ * 0.0.0.0/0 10.137.255.254 tcp dpt:8082

And lastly from sys-firewall:
$ ip route
default via 10.137.0.5 dev eth0
10.137.0.5 dev eth0 scope link

Thanks and please chime in anyone else if you have dealt with such an issue before or have any idea about what it could be.

1 Like
4

Try changing the kernel used by your sys-net AppVM to 4.x
I have difficulty connecting to any network if the sys-net AppVM is set to anything higher than 4.x.
Qube Settings → Advanced / Kernel

5

Currently sys-net kernel is version 4.14.74-1, which ships with 4.0.1.

6

Thanks for the idea, pr0xy. I did see some forum posts about kernel versions causing networking issues. But currently my sys-net kernel is version 4.14.74-1, which ships with 4.0.1.

7

I don’t have any kernels that old. I have:
4.19.155-1
4.19.152-1
Perhaps a newer kernel would help? PCI Troubleshooting | Qubes OS

8

I’m sure my 4.0.3 kernels were a lot newer.

Why are you sending PCI Troubleshooting advice? My network adapter works perfectly.

9

That shows how to get newer kernels into the Template in case that might help.

10

I don’t see anything about how to install alternative kernels in that document. Which section?

11

Sorry, meant to link to a GitHub issue.

12

Ah, thanks pr0xy.

Quick question. If my Qubes folder has no older kernels to try, how could I upgrade dom0's kernel if it can’t get internet from sys-net?

13

That would be a challenge wouldn’t it… I did not take that into account.
In my situation running 4.0.3 I have a number of 5.x kernels and a couple of 4.x kernels to play with already, so switching between them was the way to get networking functioning again. I also started from a working networking connection to the outside world that was broken by updates when the default kernel was switched from 4.x to 5.x.

14

That would be a challenge wouldn’t it… I did not take that into account.

This is one of the reasons I call this issue – which I have replicated with three fresh 4.0.x versions and one used/updated across two computers – a critical bug. It totally cripples Qubes.

I’m going to try 4.0.4, which was just released. Then maybe 4.1 if that still has this extremely serious bug. But I’m skeptical I can have a stable system with a prerelease and really need things as solid as possible.

15

I just installed 4.0.4 on my new computer (Lenovo) and it has this same bug.

What is going on?? Is it necessary to set something after a fresh install to allow VMs to get network from sys-net or what?

16

  1. Try this command in sys-net, and then in sys-firewall; is it the same result?

    curl https://1.1.1.1

    This IP is CloudFlare’s web site. If you’re more comfortable with Google, change 1.1.1.1 to 8.8.8.8.

    Is there an immediate response with HTML, or does it seem to hang?

  2. In your sys-net, the LAN seems to be in 20-bit private space (172.16.0.0/21).

    Your DNS server is in 16-bit private space (192.168.0.0/16). Did you set up the network/wifi?

    Does the file /etc/resolv.conf within sys-net also contain a 192.168 IP address?

  3. Please run sudo nft list ruleset in sys-net and sys-firewall.

  4. Please run ip link show wls7 in sys-net; feel free to change the aa:bb:cc:dd:ee:ff values to something random or redact them in your response.

17

Thank you icequbes

  1. Curling Cloudflare results in all the page HTML in sys-net. As expected, in sys-firewall it just hangs.

  2. I did not set up the internet in my location. /etc/resolv.conf in sys-net does contain a 192.168 address after nameserver.

  3. sudo nft list ruleset in sys-net:

table ip qubes-firewall {
chain forward {
type filter hook forward priority filter; policy drop;
ct state established,related accept
iifname != "vif*" accept
ip saddr 10.137.0.13 jump qbs-10-137-0-13
}

	chain qbs-10-137-0-13 {
		accept
		reject with icmp type admin-prohibited
	}
}
table ip6 qubes-firewall {
 chain forward {
		type filter hook forward priority filter; policy drop;
		ct state established,related accept
		iifname != "vif*" accept
	}
}

sudo nft list ruleset in sys-firewall

table ip qubes-firewall {
	chain forward {
		type filter hook forward priority filter; policy drop;
		ct state established,related accept
		iifname != "vif*" accept
		ip saddr 10.137.0.13 jump qbs-10-137-0-13
	}

	chain qbs-10-137-0-13 {
		accept
		reject with icmp type admin-prohibited
	}
}
table ip6 qubes-firewall {
	chain forward {
		type filter hook forward priority filter; policy drop;
		ct state established,related accept
		iifname != "vif*" accept
	}
}
  1. Result = Device "wls7" does not exist. Tried with my wifi device number and get:

2: wls6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DORMANT group default qlen 1000
link/ether rd:rd:rd:rd:rd:rd brd ff:ff:ff:ff:ff:ff

18

I moved the computer to a new location and networking between qubes works perfectly now. So it was something to do with the router, seemingly. Thank you so much for all your input.

2 Likes
19

That seems odd if that is the resolution, if the initial issue was inability to forward packets to/from sys-net-connected VMs.

I only mention this because without a definitive identification of the problem, it might return.

20

I agree completely, icequbes1. Although at the new location at least I have access to the router in question and could change settings if need be (not that I would really know what to do).

It’s very unintuitive to me that sys-net was able to connect but not share, but with a different connection it is.

I recommend asking Brendan on github on my issue SOLVED: No internet outside sys-net · Issue #6442 · QubesOS/qubes-issues · GitHub. Here is what he had to say that led me to trying the machine at a new location:

Thinking maybe a local network issue with Max hops? … Does your router/access-point have a limit in the TTL allowed for incoming local packets? Does it generate local packets with very low TTL values? … Thinking maybe a local network issue with Max hops? Does your router/access-point have a limit in the TTL allowed for incoming local packets? Does it generate local packets with very low TTL values? Have you tried the default Qubes installs on other wifi networks? Have you tried with Ethernet (assuming you have a port on the laptop(s) and that you have assigned the device to sys-net)?