Restored AppVM & Template - no networking

I recently restored a full Debian 11 template and associated AppVMs from R4.0 to a new install of R4.2

The AppVMs based on the restored template no longer have network connectivity. They can’t see the internet. Changing the NetVM makes no difference.

$ ping yahoo.com
ping: yahoo.com: Temporary failure in name resolution

$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:16:3e:5e:6c:00 brd ff:ff:ff:ff:ff:ff

I tried reinstalling the qubes-core-agent suite in case that might make a difference, but nothing needed updating on the template.

I’m unsure how to proceed.

I made a new AppVM Qube based on this template with the default `sys-firewall` as NetVM. The results are the same. No internet connectivity. Unable to ping from the terminal.

Can you provide the output of this command:

dpkg -l | grep -i qubes

I had networking issues after upgrade to 4.2, only sys-net still had network. Network was not flowing to sys-firewall, sys-whonix and so on when they were based on debian-12 and derivative templates.

Could not really find the root cause but solved the issue by reinstalling fresh debian-12 templates from their respective bash history.

Similarly, couldn’t you reinstall a fresh debian 12 template from your old one bash history ?

@DVM here is the output from a terminal in the AppVM based on the Debian 11 template:

user@test:~$ dpkg -l | grep -i qubes
ii  fwupd-qubes-vm:amd64                   1.8.14-5+deb11u1                   amd64        fwupd wrapper for Qubes OS - VM scripts
ii  libqubes-pure0                         4.2.15+deb11u1                     amd64        Qubes file copy protocol library
ii  libqubes-rpc-filecopy2                 4.2.15+deb11u1                     amd64        Qubes file copy protocol library
ii  libqubesdb                             4.2.4-1+deb11u1                    amd64        QubesDB libs.
ii  libvchan-xen1                          4.2.1-1+deb11u1                    amd64        Qubes Xen core libraries
ii  pulseaudio-qubes                       4.2.11-1+deb11u1                   amd64        Audio support for Qubes VM
ii  python3-qubesdb                        4.2.4-1+deb11u1                    amd64        QubesDB python bindings.
ii  python3-qubesimgconverter              4.2.15+deb11u1                     amd64        Python package qubesimgconverter
ii  qubes-core-agent                       4.2.27-1+deb11u1                   amd64        Qubes core agent
ii  qubes-core-agent-dom0-updates          4.2.27-1+deb11u1                   amd64        Scripts required to handle dom0 updates.
ii  qubes-core-agent-nautilus              4.2.27-1+deb11u1                   amd64        Qubes integration for Nautilus
ii  qubes-core-agent-network-manager       4.2.27-1+deb11u1                   amd64        NetworkManager integration for Qubes VM
ii  qubes-core-agent-networking            4.2.27-1+deb11u1                   amd64        Networking support for Qubes VM
ii  qubes-core-agent-passwordless-root     4.2.27-1+deb11u1                   amd64        Passwordless root access from normal user
rc  qubes-core-agent-qrexec                4.0.65-1+deb11u1                   amd64        Qubes qrexec agent
ii  qubes-core-qrexec                      4.2.16-1+deb11u1                   amd64        Qubes qrexec agent
ii  qubes-gpg-split                        2.0.70-1+deb11u1                   amd64        The Qubes service for secure gpg separation
ii  qubes-gui-agent                        4.2.11-1+deb11u1                   amd64        Makes X11 windows available to qubes dom0
ii  qubes-img-converter                    1.2.16-1+deb11u1                   amd64        Qubes service for converting untrusted images into trusted ones.
ii  qubes-input-proxy-sender               1.0.34-1+deb11u1                   amd64        Provides Simple input events proxy
ii  qubes-mgmt-salt-vm-connector           4.2.1-1+deb11u1                    all          Interface for managing VM from dom0
ii  qubes-pdf-converter                    2.1.19-1+deb11u1                   amd64        The Qubes service for converting untrusted PDF files into trusted ones
ii  qubes-repo-templates                   4.2.2-1+deb11u1                    amd64        Repository definition for Qubes OS VM template packages.
ii  qubes-usb-proxy                        1.1.5+deb11u1                      amd64        USBIP wrapper to run it over Qubes RPC connection
ii  qubes-utils                            4.2.15+deb11u1                     amd64        Qubes Linux utilities
ii  qubes-vm-dependencies                  4.2.9-1+deb11u1                    amd64        Meta package with packages required in Qubes VM
ii  qubes-vm-recommended                   4.2.9-1+deb11u1                    amd64        Meta package with packages recommended in Qubes VM
ii  qubesdb                                4.2.4-1+deb11u1                    amd64        QubesDB management tools and daemon.
ii  qubesdb-vm                             4.2.4-1+deb11u1                    amd64        QubesDB VM service.
ii  xserver-xorg-input-qubes               4.2.11-1+deb11u1                   amd64        X input driver for injecting events from qubes-gui-agent
ii  xserver-xorg-qubes-common              4.2.11-1+deb11u1                   amd64        Common functions for qubes xserver driver
ii  xserver-xorg-video-dummyqbs            4.2.11-1+deb11u1                   amd64        Dummy X video driver for qubes-gui-agent

This was not an upgrade to R4.2. It is a clean install of R4.2. I went with the default Fedora based templates for networking. Networking works fine for other Qubes. I have sys-net, sys-whonix, and multiple sys-vpns that are functioning.

The issue is with AppVMs and a debian-11 Template restored from R4.0. These are not Qubes that provide networking.

This is broken:
sys-netsys-firewall ↔ AppVM (based on deb-11)

This works:
sys-netsys-firewall ↔ AppVM (based on deb-12 or fed-38)

Can you give the result for both of these commands?

sudo systemctl status qubes-network-uplink.service
sudo systemctl status qubes-network-uplink@eth0.service

Here are the outputs:

user@test-ham:~$ sudo systemctl status qubes-network-uplink.service
● qubes-network-uplink.service - Qubes network uplink wait
     Loaded: loaded (/lib/systemd/system/qubes-network-uplink.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Wed 2024-01-10 21:59:18 JST; 10h ago
    Process: 646 ExecStart=/usr/lib/qubes/init/network-uplink-wait.sh (code=exited, status=1/FAILURE)
   Main PID: 646 (code=exited, status=1/FAILURE)
        CPU: 12ms

Jan 10 21:59:18 test-ham systemd[1]: Starting Qubes network uplink wait...
Jan 10 21:59:18 test-ham network-uplink-wait.sh[684]: A dependency job for qubes-network-uplink@eth0.service failed. See 'journalctl -xe' for details.
Jan 10 21:59:18 test-ham systemd[1]: qubes-network-uplink.service: Main process exited, code=exited, status=1/FAILURE
Jan 10 21:59:18 test-ham systemd[1]: qubes-network-uplink.service: Failed with result 'exit-code'.
Jan 10 21:59:18 test-ham systemd[1]: Failed to start Qubes network uplink wait.

user@test-ham:~$ sudo systemctl status qubes-network-uplink@eth0.service
● qubes-network-uplink@eth0.service - Qubes network uplink (eth0) setup
     Loaded: loaded (/lib/systemd/system/qubes-network-uplink@.service; static)
     Active: inactive (dead)

Jan 10 21:59:18 test-ham systemd[1]: Dependency failed for Qubes network uplink (eth0) setup.
Jan 10 21:59:18 test-ham systemd[1]: qubes-network-uplink@eth0.service: Job qubes-network-uplink@eth0.service/start failed with result 'dependency'.

Can you run this to get more details:

sudo journalctl -u qubes-network-uplink.service -b --no-pager
user@test-ham:~$ sudo journalctl -u qubes-network-uplink.service -b --no-pager
-- Journal begins at Mon 2022-03-28 10:45:14 JST, ends at Thu 2024-01-11 08:07:45 JST. --
Jan 10 21:59:18 test-ham systemd[1]: Starting Qubes network uplink wait...
Jan 10 21:59:18 test-ham network-uplink-wait.sh[684]: A dependency job for qubes-network-uplink@eth0.service failed. See 'journalctl -xe' for details.
Jan 10 21:59:18 test-ham systemd[1]: qubes-network-uplink.service: Main process exited, code=exited, status=1/FAILURE
Jan 10 21:59:18 test-ham systemd[1]: qubes-network-uplink.service: Failed with result 'exit-code'.
Jan 10 21:59:18 test-ham systemd[1]: Failed to start Qubes network uplink wait.

Same for the other one:

sudo journalctl -u qubes-network-uplink@eth0.service -b --no-pager
user@test-ham:~$ sudo journalctl -u qubes-network-uplink@eth0.service -b --no-pager
-- Journal begins at Mon 2022-03-28 10:45:14 JST, ends at Thu 2024-01-11 08:10:47 JST. --
Jan 10 21:59:18 test-ham systemd[1]: Dependency failed for Qubes network uplink (eth0) setup.
Jan 10 21:59:18 test-ham systemd[1]: qubes-network-uplink@eth0.service: Job qubes-network-uplink@eth0.service/start failed with result 'dependency'.

Well, no verbose logs unfortunately.
What about this service:

sudo journalctl -u networking -b --no-pager
user@test-ham:~$ sudo journalctl -u networking -b --no-pager
-- Journal begins at Mon 2022-03-28 10:45:14 JST, ends at Thu 2024-01-11 08:19:44 JST. --
Jan 10 21:59:18 test-ham systemd[1]: Starting Raise network interfaces...
Jan 10 21:59:18 test-ham systemd[1]: Finished Raise network interfaces.

Just to make sure, what happens if you start the service manually?

sudo systemctl start qubes-network-uplink.service
user@test-ham:~$ sudo systemctl start qubes-network-uplink.service
Job for qubes-network-uplink.service failed because the control process exited with error code.
See "systemctl status qubes-network-uplink.service" and "journalctl -xe" for details.

Can you get the full boot log and upload it here?

sudo journalctl -b --no-pager > boot.log

@DVM The file was too big to upload here.

I placed it here temporarily:

The following logs occurred when you manually started the service earlier:

Jan 11 08:22:08 test-ham sudo[32499]:     user : TTY=pts/0 ; PWD=/home/user ; USER=root ; COMMAND=/bin/systemctl start qubes-network-uplink.service
Jan 11 08:22:08 test-ham sudo[32499]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1000)
Jan 11 08:22:08 test-ham systemd[1]: Starting Qubes network uplink wait...
Jan 11 08:22:08 test-ham systemd[1]: Starting Qubes base firewall settings...
Jan 11 08:22:08 test-ham qubes-iptables[32513]: nft: /sbin/nft does not exist.
Jan 11 08:22:08 test-ham systemd[1]: qubes-iptables.service: Main process exited, code=exited, status=5/NOTINSTALLED
Jan 11 08:22:08 test-ham systemd[1]: qubes-iptables.service: Failed with result 'exit-code'.
Jan 11 08:22:08 test-ham systemd[1]: Failed to start Qubes base firewall settings.
Jan 11 08:22:08 test-ham systemd[1]: Dependency failed for Network (Pre).
Jan 11 08:22:08 test-ham systemd[1]: Dependency failed for Qubes network uplink (eth0) setup.
Jan 11 08:22:08 test-ham systemd[1]: qubes-network-uplink@eth0.service: Job qubes-network-uplink@eth0.service/start failed with result 'dependency'.
Jan 11 08:22:08 test-ham systemd[1]: network-pre.target: Job network-pre.target/start failed with result 'dependency'.
Jan 11 08:22:08 test-ham network-uplink-wait.sh[32512]: A dependency job for qubes-network-uplink@eth0.service failed. See 'journalctl -xe' for details.
Jan 11 08:22:08 test-ham systemd[1]: qubes-network-uplink.service: Main process exited, code=exited, status=1/FAILURE
Jan 11 08:22:08 test-ham systemd[1]: qubes-network-uplink.service: Failed with result 'exit-code'.
Jan 11 08:22:08 test-ham systemd[1]: Failed to start Qubes network uplink wait.

There is one line here that could be the cause of this “dependency” error:

Jan 11 08:22:08 test-ham qubes-iptables[32513]: nft: /sbin/nft does not exist.

Can you check if you have nftables installed in the template?

dpkg -l | grep nftables

It looks like nftables are present

user@test-ham:~$ dpkg -l | grep nftables
ii  libnftables1:amd64                     0.9.8-3.1+deb11u1                  amd64        Netfilter nftables high level userspace API library
ii  libnftnl11:amd64                       1.1.9-1                            amd64        Netfilter nftables userspace API library
ii  nftables                               0.9.8-3.1+deb11u1                  amd64        Program to control packet filtering rules by Netfilter project

Can you find it in the same path in the template?

ls -l /sbin/nft