Preloaded dispVMs losing network connectivity

User Support Testing Release
1

Noticed a strange thing: if I do not run new dispVMs for a while, “old” preloaded dispVMs appear with non-functional networking. I can ping internal qubes network, but outside connectivity does not work – seemingly NAT rules for them expire somehow.

2

Did you do anything special in the meantime, like suspend / resume, changing a netvm, etc… ?

3

apparently, no! however, could restart netvm. will try to reproduce.

4

Please ping me once you can reproduce it (I won’t get notified otherwise).

Also, don’t turn off the disposables once that happens, I will try to provide guidance once that happens.

For now, I’d say, when that happens:

Are you using IPv6?

On the disposable that was preloaded:

ip r
ip -4 a show dev eth0
# if the last command doesn't show enough
ip a

On the netvm:

ip r

Can you see the IP of the disposable in the netvm?

5

My uplink is v4-only. Did not manage to reproduce the bug yet.

6

This might fix it:

3 Likes
7

Has this worked its way into rc3 because I have a similar problem but only when a restart, after an upgrade, is needed. After an upgrade and a restart the remaining (in my case 2) preloaded disposables - once started will not connect to the internet.

8

Yes. This seems like a different issue.

Can you explain in steps this again?

  1. Configured to preload 2 disposable
  2. Updated using the Updater GUI
  3. In the end, when it shuts down the template, it restart the preloaded disposables, but attempting to use them result in lack of networking

This is what I understand, please add more details if necessary. Can you attempt the same using a disposable template without preloaded disposables?

  1. Clone default-dvm to default-dvm-test
  2. Disable preloading (set to 0) for default-dvm-test
  3. Update a template
  4. Check if there is networking on disposables based of each disposable template:
    1. qvm-run -p --dispvm=default-dvm -- 'ip r; ping qubes-os.org'
    2. qvm-run -p --dispvm=default-dvm-test -- 'ip r; ping qubes-os.org'
    3. You may also launch Firefox from the app menu to test networking, but ip r still has to be run from the terminal
  5. On another terminal in dom0, watch the qubesd logs, it might tell if there was a net device attach/detach failure: sudo journalctl -fu qubesd
9

So to the first part of your answer

Its actually the resarting of sys-firewall, sys-net, syd-usb that triggers the issue. So on starting the next x preloaded disposable qube it cannot access the internet. Normal non disposable qubes will connect to the internet. It seems that after the pre defined preloaded qubes are flushed out then networking restarts on the next disposable that are started.

HTH

Will try the second part and reply asap

10

default-dvm does not have networking
default-dvm-test has networking

WARNING: vm.disp7295: Deferred attaching libvirt net device because qube is paused

I get one of the above warnings for every preloaded qube.

HTH

11

Confirmed. Can you please open an issue on Github?

  • ip r or ip a doesn’t show any issue
  • the qube has the correct gateway in qubesdb, but the gateway has not registered the client in the routing table
  • after disconnecting netvm property (setting to ‘’) and reverting to the default of sys-firewall, the gateway routing table is updated and the qube can connect

@marmarek have you seen this?

This is fine. You may also see vm.disp7295: Deferred detaching ....

12

The easiest way to get to the issue is:

  • Preload
  • Restart its netvm
  • Try to ping

I believe the issue is, when it connects to the same netvm it already has, it doesn’t try to reattach because I was thinking of switching between netvms and then switching back to the original netvm. But in this case, because the netvm shuts down, it clears the routing table, and as it doesn’t try to reatach, the netvm is not notified that the client exists.

13

Seems to be related to never setting the deferred-netvm-original as it is only called if the netvm property has changed.

14

could be related to Apply fix for quick network reconnect after backend destroy by marmarek · Pull Request #1193 · QubesOS/qubes-linux-kernel · GitHub maybe? The patch wasn’t merged upstream yet.

15

After trying a few combinations it seems for me at least, that it is not restarting sys-net that causes the issue, it is restarting sys-firewall.

16

@Danan

The issue was the deferred netvm feature was not being set on current netvm shutdown. This should fix it for current netvm shutdown or restart.