[qubes-users] Update Killed All Non-Tor Internet Access!

Mailing Lists qubes-users
1

After updating dom0 and all my templatevms yesterday and rebooting, I can now no longer access the internet except through VMs which have my torvm as their netvm. (Of course, my netvm itself can connect to the internet, which I verified with the terminal.)

I'm pretty sure it was the update, since everything was working fine right before rebooting to apply the update. I didn't change anything else.

I tried creating a new, plain firewallvm to connect through. No joy.

I'm sure I'm not the only one experiencing this, as my setup is pretty standard. Any idea what could be causing this?

2

My templatevm's yum.log shows that the following were updated (note that this is just a vanilla Qubes template; I've never installed anything there myself):

Aug 15 03:37:22 Updated: 1:NetworkManager-glib-0.9.8.2-1.fc18.x86_64
Aug 15 03:37:23 Updated: selinux-policy-3.11.1-100.fc18.noarch
Aug 15 03:37:23 Updated: gstreamer1-1.0.9-1.fc18.x86_64
Aug 15 03:37:24 Updated: gstreamer1-plugins-base-1.0.9-1.fc18.x86_64
Aug 15 03:37:24 Updated: libnm-gtk-0.9.8.2-1.fc18.x86_64
Aug 15 03:37:25 Updated: 2:libwbclient-4.0.8-1.fc18.x86_64
Aug 15 03:37:26 Updated: 2:samba-libs-4.0.8-1.fc18.x86_64
Aug 15 03:37:27 Updated: 2:samba-common-4.0.8-1.fc18.x86_64
Aug 15 03:37:27 Updated: 2:libsmbclient-4.0.8-1.fc18.x86_64
Aug 15 03:37:28 Updated: nm-connection-editor-0.9.8.2-1.fc18.x86_64
Aug 15 03:37:29 Updated: 1:NetworkManager-0.9.8.2-1.fc18.x86_64
Aug 15 03:37:30 Updated: hplip-common-3.13.7-1.fc18.x86_64
Aug 15 03:37:31 Updated: hplip-libs-3.13.7-1.fc18.x86_64
Aug 15 03:37:32 Updated: 1:hpijs-3.13.7-1.fc18.x86_64
Aug 15 03:37:35 Updated: xulrunner-23.0-2.fc18.x86_64
Aug 15 03:37:36 Updated: paps-libs-0.6.8-26.fc18.x86_64
Aug 15 03:37:37 Updated: paps-0.6.8-26.fc18.x86_64
Aug 15 03:37:40 Updated: firefox-23.0-1.fc18.x86_64
Aug 15 03:37:41 Updated: hplip-3.13.7-1.fc18.x86_64
Aug 15 03:37:41 Updated: libsane-hpaio-3.13.7-1.fc18.x86_64
Aug 15 03:37:42 Updated: network-manager-applet-0.9.8.2-1.fc18.x86_64
Aug 15 03:37:42 Updated: 2:samba-client-4.0.8-1.fc18.x86_64
Aug 15 03:37:43 Updated: gstreamer1-plugins-good-1.0.9-1.fc18.x86_64
Aug 15 03:37:43 Updated: gstreamer1-plugins-bad-free-1.0.9-1.fc18.x86_64
Aug 15 03:37:44 Updated: selinux-policy-targeted-3.11.1-100.fc18.noarch
Aug 15 03:37:45 Updated: selinux-policy-devel-3.11.1-100.fc18.noarch
Aug 15 03:37:46 Updated: selinux-policy-doc-3.11.1-100.fc18.noarch
Aug 15 03:37:47 Updated: binutils-2.23.51.0.1-10.fc18.x86_64
Aug 15 03:37:48 Updated: gnupg-1.4.14-1.fc18.x86_64
Aug 15 03:37:48 Updated: boost-date-time-1.50.0-7.fc18.x86_64
Aug 15 03:37:53 Updated: thunderbird-17.0.8-1.fc18.x86_64

And dom0's yum.log shows the following were updated yesterday:

2:libwbclient-4.0.8-1.fc18.x86_64
2:samba-libs-4.0.8-1.fc18.x86_64
2:samba-common-4.0.8-1.fc18.x86_64
system-config-keyboard-base-1.3.1-14.fc18.x86_64
system-config-keyboard-1.3.1-14.fc18.x86_64
2:libsmbclient-4.0.8-1.fc18.x86_64
boost-program-options-1.50.0-7.fc18.x86_64
perf-3.10.6-100.fc18.x86_64
binutils-2.23.51.0.1-10.fc18.x86_64
gstreamer1-1.0.9-1.fc18.x86_64

Could it be the NetworkManager updates in the templatevm? I'm going to try to figure out if/how I can try rolling those back.

3

Did you try to reboot the whole system? Perhaps it was some restarted
VMs not reconnecting to the net/firewallvms (issue #722)?

j.

4

If you only use one default template, then surely no.

If your can acces any network from your netVM, then your template (and
dom0) is fine.

1. Try to reassign the the netvm for your AppVM first...
2. check the firewall settings (if it's any)
3. dump your traffic on the netVM to see if AppVM traffic is going to
the right direction...

5

Yeah, I rebooted several times earlier when I thought it might be a DNS problem. Changed the DNS server settings in NetworkManager, but that didn't affect it.

6

Could it be the NetworkManager updates in the templatevm? I'm going to try
to figure out if/how I can try rolling those back.

If you only use one default template, then surely no.

The torvm has its own template, but everything else (including netvm and firewallvm) uses the default template.

If your can acces any network from your netVM, then your template (and
dom0) is fine.

Actually, I just discovered that my templatevms can connect to the update servers over my plain connection. (Will expand on this in another message.)

1. Try to reassign the the netvm for your AppVM first...

Ok, I tried changing some AppVM's NetVM to netvm (instead of the usual firewallvm). No change.

Also, as said above, I tried creating a new firewallvm (connected to netvm), and assigning that as the NetVM for some AppVMs, with no change.

2. check the firewall settings (if it's any)

Right, no firewall settings. (Anyway, I checked all the AppVMs, some of which have firewall settings, and some of which don't.)

3. dump your traffic on the netVM to see if AppVM traffic is going to
the right direction...

Running sudo tcpdump in the netvm, it looks like traffic is going in the right direction. But I'm not sure if I'm reading it correctly, and there are some other odd things in there that I don't understand. I attached a partial output. Would you mind taking a look? (For the output, I just started tcpdump, then tried to connect to qubes-os.org with a dispvm.)

One thing I noticed in the tcpdump output is a lot of lines like this:

06:17:44.271406 IP netvm.38028 > 10.137.2.1.domain: 7512+ A? qubes-os.org. (30)
06:17:44.271418 IP netvm.38028 > 10.137.2.1.domain: 49126+ AAAA? qubes-os.org. (30)
06:17:49.273462 IP netvm.53980 > 10.137.2.254.domain: 7512+ A? qubes-os.org. (30)
06:17:49.273509 IP netvm.53980 > 10.137.2.254.domain: 49126+ AAAA? qubes-os.org. (30)

Doesn't the "AAAA" stuff have to do with IPv6? I'm pretty sure my equipment only handles IPv4, but I'm not sure if this is normal.

(Attachment tcpdump.txt is missing)

7

I'll start by saying sorry for the long message. I've tried to <snip> where possible to exclude extraneous things. :frowning:
As I mentioned in a previous message, I tried to see if I can rollback the updates to NetworkManager:

[user@fedora-18-x64-clone-1 ~]$ sudo yum downgrade NetworkManager*
Loaded plugins: langpacks, post-transaction-actions, presto, refresh-packagekit, yum-qubes-hooks
No Match for available package: 1:NetworkManager-devel-0.9.7.0-12.git20121004.fc18.i686
No Match for available package: 1:NetworkManager-devel-0.9.7.0-12.git20121004.fc18.x86_64
No Match for available package: 1:NetworkManager-glib-devel-0.9.7.0-12.git20121004.fc18.i686
No Match for available package: 1:NetworkManager-glib-devel-0.9.7.0-12.git20121004.fc18.x86_64
No Match for available package: NetworkManager-l2tp-0.9.6-2.fc18.x86_64
No Match for available package: NetworkManager-openswan-0.9.3.995-3.git20120302.fc18.x86_64
No Match for available package: NetworkManager-ssh-0.0.3-0.8.20130419git3d5321b.fc18.x86_64
No Match for available package: NetworkManager-ssh-gnome-0.0.3-0.8.20130419git3d5321b.fc18.x86_64
No Match for available package: 1:NetworkManager-wimax-0.9.7.0-12.git20121004.fc18.x86_64
Resolving Dependencies
--> Running transaction check
---> Package NetworkManager.x86_64 1:0.9.7.0-12.git20121004.fc18 will be a downgrade
---> Package NetworkManager.x86_64 1:0.9.8.2-1.fc18 will be erased
---> Package NetworkManager-glib.x86_64 1:0.9.7.0-12.git20121004.fc18 will be a downgrade
---> Package NetworkManager-glib.x86_64 1:0.9.8.2-1.fc18 will be erased
--> Finished Dependency Resolution
Error: Package: nm-connection-editor-0.9.8.2-1.fc18.x86_64 (@updates)
            Requires: NetworkManager-glib >= 1:0.9.8.0
            Removing: 1:NetworkManager-glib-0.9.8.2-1.fc18.x86_64 (@updates)
                NetworkManager-glib = 1:0.9.8.2-1.fc18
            Downgraded By: 1:NetworkManager-glib-0.9.7.0-12.git20121004.fc18.x86_64 (fedora)
                NetworkManager-glib = 1:0.9.7.0-12.git20121004.fc18
Error: Package: network-manager-applet-0.9.8.2-1.fc18.x86_64 (@updates)
            Requires: NetworkManager-glib >= 1:0.9.8.0
            Removing: 1:NetworkManager-glib-0.9.8.2-1.fc18.x86_64 (@updates)
                NetworkManager-glib = 1:0.9.8.2-1.fc18
            Downgraded By: 1:NetworkManager-glib-0.9.7.0-12.git20121004.fc18.x86_64 (fedora)
                NetworkManager-glib = 1:0.9.7.0-12.git20121004.fc18
Error: Package: network-manager-applet-0.9.8.2-1.fc18.x86_64 (@updates)
            Requires: NetworkManager >= 1:0.9.8.0
            Removing: 1:NetworkManager-0.9.8.2-1.fc18.x86_64 (@updates)
                NetworkManager = 1:0.9.8.2-1.fc18
            Downgraded By: 1:NetworkManager-0.9.7.0-12.git20121004.fc18.x86_64 (fedora)
                NetworkManager = 1:0.9.7.0-12.git20121004.fc18
  You could try using --skip-broken to work around the problem
  You could try running: rpm -Va --nofiles --nodigest

Just for the sake of completeness, I tried both of the suggestions above, which didn't work.

I then did some more searching and learned that there is a handy way to undo yum changes in Fedora using yum history.[1][2] First, I ran "sudo yum history" to find the ID of the update. For me it was 16. Then I ran sudo yum history undo 16:

[user@fedora-18-x64-clone-1 ~]$ sudo yum history undo 16
Loaded plugins: langpacks, post-transaction-actions, presto, refresh-packagekit, yum-qubes-hooks
Undoing transaction 16, from Thu Aug 15 03:37:21 2013
     Updated NetworkManager-1:0.9.8.1-3.git20130514.fc18.x86_64 @updates
     Update 1:0.9.8.2-1.fc18.x86_64 @updates
     Updated NetworkManager-glib-1:0.9.8.1-3.git20130514.fc18.x86_64 @updates
     Update 1:0.9.8.2-1.fc18.x86_64 @updates

     <snip>

     Updated network-manager-applet-0.9.8.1-4.git20130514.fc18.x86_64 @updates
     Update 0.9.8.2-1.fc18.x86_64 @updates
     Updated nm-connection-editor-0.9.8.1-4.git20130514.fc18.x86_64 @updates
     Update 0.9.8.2-1.fc18.x86_64 @updates

     <snip>

Failed to downgrade: 1:NetworkManager-0.9.8.1-3.git20130514.fc18.x86_64
Failed to downgrade: 1:NetworkManager-glib-0.9.8.1-3.git20130514.fc18.x86_64

<snip>

Failed to downgrade: network-manager-applet-0.9.8.1-4.git20130514.fc18.x86_64
Failed to downgrade: nm-connection-editor-0.9.8.1-4.git20130514.fc18.x86_64

<snip>

Dependencies Resolved

8

Ok, so my system is set up as follows:

NetVM: netvm
ProxyVM 1: firewallvm
ProxyVM 2: torvm
TemplateVM 1: fedora-18-x64 (vanilla default)
TemplateVM 1: fedora-18-x64-net (standard Qubes Tor installation)
AppVM 1: untrusted (NetVM: firewallvm)
AppVM 2: untrusted-tor (NetVM: torvm)
[A bunch of other AppVMs which all use firewallvm as their NetVM.]

And I've established the following with respect to connectivity:

netvm: can connect
torvm: can connect (my tests from inside the VM didn't work, but it obviously must be able to, since AppVMs which use it can connect successfully)
firewallvm: ??? (when I issue "host google.com", it just times out, but when I issue "ping 173.194.66.102", it works?!)
fedora-18-x64: can communicate with the update servers
fedora-18-x64-net: can communicate with the update servers
untrusted: cannot connect
untrusted-tor: can connect (http, pop3, smtp all work)

So, what I'm having trouble figuring out is whether the problem is with the firewallvm, or something else.

It seems like a DNS issue, but I already changed my DNS several times (and of course rebooted the whole system after each change). I tried my own ISPs DNS server, OpenDNs, and Google Public DNS. I had it set to OpenDNS before (when everything was working correctly) because my ISP's DNS was causing problems (an unrelated Windows machine on my local network wasn't able to connect at all a couple days ago using my ISP's DNS until I changed it to OpenDNS). I wonder if somehow the update to NetworkManager broke my ability to set the DNS server or something.

9

Sorry for sending so many messages, but I just wanted to add one more clue:

I have a VM that I use to access my router's administration page. It uses firewallvm as its NetVM, and it's fine (i.e. I can connect to the router and everything as normal).

I should also add that I have other computers here which have been connected normally the whole time (some Windows, some Linux, including Fedora).

Also, I know this is rather obvious from my previous messages, but if I *switch* an AppVMs NetVM from firewallvm to torvm, then I can connect successfully (through tor, of course!).

Lastly, I mentioned some DNS-related things in a previous message, but I want to add that a computer I have here running plain Fedora 19 has not had its DNS settings changed at all. It shows that it's still using my default ISP DNS, and its connection is fine.

So, given all of the information above and in the previous messages, I think I can confidently say:

It's not a hardware problem.
It's not an ISP problem.
It's not a NetVM problem.
It's not an AppVM problem.
It doesn't *seem* like a ProxyVM problem. (But what else could it be?!)

This is a really strange phenomenon, right?? I mean, how is this even possible?

Or maybe it only seems really strange to me because there are things I am ignorant about or do not understand. :slight_smile:

10

I'm afraid I can't help you, but I had the exact same behaviour yesterday.
After what was a massive update for some reason (even though I always
update the template when I see the notification), I rebooted the machine
and then realised I had no network.

Tried a couple of things with tshark/tcpdump on netvm/firewallvm vif and
eth0 interfaces, gave myself a headache trying to remember iptables rules
etc, had fun reading about "network administratively prohibited" and "host
administratively prohibited" ICMP Destination Unreachable messages (which
was what my sniffer were showing me) and then I noticed that it was a name
resolution issue. Hardcoding the IP of my local (physical) router (which
runs a DNS forwarder) in /etc/resolv.conf would make the AppVM resolve
names.

So I thought fine, fired up the templateVM and hardcoded the IP of my
router there, and then rebooted. The machine appeared to work fine after
the reboot and I thought I was very clever, until I noticed that
/etc/resolv.conf of my active VMs had been returned to their
pre-intervention values (probably by NetworkManager) of
nameserver 10.137.2.1
nameserver 10.137.2.254

So, no idea wtf happened, but it all works now. :frowning:

Alex

11

Just had what I assume is same problem (no network in AppVMs after a main template update), except that for me even net access via TorVM didn’t work I had to cure by restoring template from backup before i did, I notice that in a Terminal in the NetVM I could ping websites by name (i.e. DNS working), however …in the FirewallVM I could ping by IP address but not by name (DNS not working) hope helps - I still have the bad template if want me run a any debugs CB – You received this message because you are subscribed to the Google Groups “qubes-users” group. To unsubscribe from this group and stop receiving emails from it, send an email to qubes-users+unsubscribe@googlegroups.com. To post to this group, send email to qubes-users@googlegroups.com. Visit this group at . For more options, visit .

12

The DNS traffic is DNATed at each netvm/firewallvm. So each VM's DNS points at
its "uplink" VM magic IP, which is redirected to the next VM (or outside
network in final netvm).
So start with checking if it is done correctly in your netvm:
sudo iptables -t nat -nvL PR-QBS

It should point to your real DNS. If not, try:
sudo /usr/lib/qubes/qubes-setup-dnat-to-ns
(it should copy settings from /etc/resolv.conf).

13

<snip>

So, given all of the information above and in the previous messages, I

think I can confidently say:

It's not a hardware problem.
It's not an ISP problem.
It's not a NetVM problem.
It's not an AppVM problem.
It doesn't *seem* like a ProxyVM problem. (But what else could it be?!)

I'm afraid I can't help you, but I had the exact same behaviour yesterday.
After what was a massive update for some reason (even though I always
update the template when I see the notification), I rebooted the machine
and then realised I had no network.

Tried a couple of things with tshark/tcpdump on netvm/firewallvm vif and
eth0 interfaces, gave myself a headache trying to remember iptables rules
etc, had fun reading about "network administratively prohibited" and "host
administratively prohibited" ICMP Destination Unreachable messages (which
was what my sniffer were showing me) and then I noticed that it was a name
resolution issue. Hardcoding the IP of my local (physical) router (which
runs a DNS forwarder) in /etc/resolv.conf would make the AppVM resolve
names.

So I thought fine, fired up the templateVM and hardcoded the IP of my
router there, and then rebooted. The machine appeared to work fine after
the reboot and I thought I was very clever, until I noticed that
/etc/resolv.conf of my active VMs had been returned to their
pre-intervention values (probably by NetworkManager) of
nameserver 10.137.2.1
nameserver 10.137.2.254

So, no idea wtf happened, but it all works now. :frowning:

Thank you so much for replying! I was beginning to think that I was going insane. :slight_smile:

The DNS traffic is DNATed at each netvm/firewallvm. So each VM's DNS points at
its "uplink" VM magic IP, which is redirected to the next VM (or outside
network in final netvm).
So start with checking if it is done correctly in your netvm:
sudo iptables -t nat -nvL PR-QBS

It should point to your real DNS. If not, try:
sudo /usr/lib/qubes/qubes-setup-dnat-to-ns
(it should copy settings from /etc/resolv.conf).

It worked! Marek saves the day again! :slight_smile:

Just a couple of minor points to help others with the same issue:

The file is actually qubes_setup_dnat_to_ns (at least on my system).
Also, I think you accidentally left out "sh." So, full command: "sudo sh qubes_setup_dnat_to_ns".
Lastly, running this command changed the destination IP address range to the range where my firewallvm lives (not the external DNS server IP addresses, which are in /etc/resolv.conf).

Thank you, Marek! :slight_smile:

14

hear hear :slight_smile: Although I got my network back through restoring the main template from backup, I am still wondering how to prevent recurrence when I next update (BTW I hadn’t noticed but discovered the reason my TorVM wasn’t working was but have now fixed) do we know which package caused the problem in the update (or why?) CB – You received this message because you are subscribed to the Google Groups “qubes-users” group. To unsubscribe from this group and stop receiving emails from it, send an email to qubes-users+unsubscribe@googlegroups.com. To post to this group, send email to qubes-users@googlegroups.com. Visit this group at . For more options, visit .

15

So, after I reboot, I had to apply this again. Maybe I need to do it in the template upon which netvm is based to get it to stick across reboots?

Anyway, I think I was mistaken about what it did and did not change, since it looks like it does indeed cause the "udp dpt" portion of the iptables output to point to my real external proxy. So, sorry about that. :slight_smile:

A similar (perhaps related?) issue: I've noticed that when I reboot the whole machine, I have to start up torvm, then shut it down, then start it again, before connections to the Tor network can actually be made. Not sure why, and obviously this is not at all a difficult workaround, but I thought I would share the data point.

16

I haven't observed such behavior on my system (last template update two days
ago or so)...
Perhaps NetworkManager no longer call this script after connecting to the
network (it is called from /etc/NetworkManager/dispatcher.d/qubes-nmhook).
Check the logs in netvm - perhaps there is some hint...

17

established trial & error the prob in the template update is in NetworkManager

CB

18

I copied logs from:

  • BAD (updated NetworkManager - network not working in AppVMs) and
  • GOOD (template not updated about a month - network working fine)

…versions of NetVM into attached LibreOffice spreadsheet (separate sheets)

but could not see what the problem might be ?

CB

(Attachment Good and Bad (NetworkManager) NetVM logs.ods is missing)

19

I can report that the problem recurs on my system whenever the netvm is shut down and started back up. (In other words, not just when dom0 is restarted, but also after, e.g., running qvm-backup.) I have to apply Marek's fix, then restart all the ProxyVMs in order to get a connection in any AppVMs. (Oddly, it now appears that this is true of Tor-connected VMs, as well. As I relayed in my previous messages, this was not the case before.)

20

It looks to be the case of:

For now the workaround (until Fedora releases the fix) is to do in template:
sudo systemctl enable NetworkManager-dispatcher.service