Can somebody clarifies how qubes-update-check service works and how dom0 gets update notifications for TemplateVMs that are never powered on?

Insurgo · October 12, 2022, 3:47pm

I’m so sorry for the confusion here, @AndyZ is totally right: -d is putting service off, -D puts service into its default status.

Thanks to @ludovic for his awesome explanation, giving implementation details:

ludovic:

The AppVm checks for the available updates.

In the AppVM , five minutes after the boot, qubes-update-check.timer launches qubes-update-check.service which checks for available updates and then notifies dom0 that update are (not) available for the TemplateVM .

Example with the test-f35 App-VM based on the fedora-35 TemplateVM

[user@test-f35 ~]$ sudo systemctl list-units| grep update
  systemd-update-utmp.service                       loaded active exited    Record System Boot/Shutdown in UTMP
  qubes-update-check.timer                          loaded active waiting   Periodically check for updates
  unbound-anchor.timer                              loaded active waiting   daily update of the root trust anchor for DNSSEC

[user@test-f35 ~]$ systemctl cat qubes-update-check.timer
# /usr/lib/systemd/system/qubes-update-check.timer
[Unit]
Description=Periodically check for updates
ConditionPathExists=/var/run/qubes-service/qubes-update-check

[Timer]
OnBootSec=5min
OnUnitActiveSec=2d

[Install]
WantedBy=multi-user.target

[user@test-f35 ~]$ systemctl cat qubes-update-check.service
# /usr/lib/systemd/system/qubes-update-check.service
[Unit]
Description=Qubes check for VM updates and notify dom0
ConditionPathExists=/var/run/qubes-service/qubes-update-check
After=qubes-qrexec-agent.service

[Service]
Type=oneshot
ExecStart=-/usr/lib/qubes/upgrades-status-notify

[user@test-f35 ~]$ cat /usr/lib/qubes/upgrades-status-notify
#!/usr/bin/bash

set -e

upgrades_installed="$(/usr/lib/qubes/upgrades-installed-check)"

if [ "$upgrades_installed" = "true" ]; then
    /usr/lib/qubes/qrexec-client-vm dom0 qubes.NotifyUpdates /bin/sh -c 'echo 0'
elif [ "$upgrades_installed" = "false" ]; then
    /usr/lib/qubes/qrexec-client-vm dom0 qubes.NotifyUpdates /bin/sh -c 'echo 1'
fi

[user@test-f35 ~]$ cat /usr/lib/qubes/upgrades-installed-check
#!/usr/bin/bash

## `echo`s:
## * 'true' - if all upgrades have been installed
## * 'false' - if there are pending upgrades
## * nothing - if apt-get is currently locked
##
## Forwards the exit code of the package manager.

if [ -e /etc/system-release ]; then
    ## Fedora
    if command -v dnf >/dev/null; then
        yum=dnf
    else
        yum=yum
    fi
    # shellcheck disable=SC2034
    yum_output="$($yum -yq check-update)"
    exit_code="$?"
    [ "$exit_code" -eq 100 ] && echo "false" && exit 0
    [ "$exit_code" -eq 0 ] && echo "true"
elif [ -e /etc/debian_version ]; then

[...]

fi

exit "$exit_code"

Now let’s test with vault (an example of an offline (airgapped) qube, which is expected to not get online at all:

[user@dom0 ~]$ qvm-service vault
[user@dom0 ~]$

No customized services enabled.

Let’s check journals for service traces:

[user@vault ~]$ sudo journalctl --boot  0 | grep -e updates
Oct 12 11:04:21 vault systemd[1]: Started qubes-update-check.timer - Periodically check for updates.
Oct 12 11:04:21 vault systemd[1]: qubes-updates-proxy-forwarder.socket - Forward connection to updates proxy over Qubes RPC was skipped because of a failed condition check (ConditionPathExists=/var/run/qubes-service/updates-proxy-setup).
Oct 12 11:04:21 vault systemd[1]: qubes-updates-proxy.service - Qubes updates proxy (tinyproxy) was skipped because all trigger condition checks failed.

Update check enabled, but no proxy setup enabled.

Using shaker/cacher here (but default tinyproxy is just different exposed update proxy for AppVM, so just dismiss http://HTTPS/// prepending URLs below):

[user@vault ~]$ sudo dnf update
Fedora 36 openh264 (From Cisco) - x86_64        0.0  B/s |   0  B     00:00    
Errors during downloading metadata for repository 'fedora-cisco-openh264':
  - Curl error (6): Couldn't resolve host name for http://HTTPS///mirrors.fedoraproject.org/metalink?repo=fedora-cisco-openh264-36&arch=x86_64&protocol=http&protocol=http [Could not resolve host: HTTPS]
Error: Failed to download metadata for repo 'fedora-cisco-openh264': Cannot prepare internal mirrorlist: Curl error (6): Couldn't resolve host name for http://HTTPS///mirrors.fedoraproject.org/metalink?repo=fedora-cisco-openh264-36&arch=x86_64&protocol=http&protocol=http [Could not resolve host: HTTPS]
Fedora 36 - x86_64 - Updates                    0.0  B/s |   0  B     00:00    
Errors during downloading metadata for repository 'updates':
  - Curl error (6): Couldn't resolve host name for http://HTTPS///mirrors.fedoraproject.org/metalink?repo=updates-released-f36&arch=x86_64&protocol=http&protocol=http [Could not resolve host: HTTPS]
Error: Failed to download metadata for repo 'updates': Cannot prepare internal mirrorlist: Curl error (6): Couldn't resolve host name for http://HTTPS///mirrors.fedoraproject.org/metalink?repo=updates-released-f36&arch=x86_64&protocol=http&protocol=http [Could not resolve host: HTTPS]

No with defaults service setting, qubes update check enabled but update proxy setup disable, update check fails (no network access succeeds, no socket enabled for update check to succeed).

Now let’s change only the update proxy setup service and see outcome:

[user@dom0 ~]$ qvm-shutdown vault
[user@dom0 ~]$ qvm-service vault updates-proxy-setup off
[user@dom0 ~]$ qvm-service vault
updates-proxy-setup  off
[user@dom0 ~]$ qvm-run --pass-io --no-gui vault "sudo dnf update"
Fedora 36 openh264 (From Cisco) - x86_64        0.0  B/s |   0  B     00:00    
Errors during downloading metadata for repository 'fedora-cisco-openh264':
  - Curl error (6): Couldn't resolve host name for http://HTTPS///mirrors.fedoraproject.org/metalink?repo=fedora-cisco-openh264-36&arch=x86_64&protocol=http&protocol=http [Could not resolve host: HTTPS]
Error: Failed to download metadata for repo 'fedora-cisco-openh264': Cannot prepare internal mirrorlist: Curl error (6): Couldn't resolve host name for http://HTTPS///mirrors.fedoraproject.org/metalink?repo=fedora-cisco-openh264-36&arch=x86_64&protocol=http&protocol=http [Could not resolve host: HTTPS]
Fedora 36 - x86_64 - Updates                    0.0  B/s |   0  B     00:00    
Errors during downloading metadata for repository 'updates':
  - Curl error (6): Couldn't resolve host name for http://HTTPS///mirrors.fedoraproject.org/metalink?repo=updates-released-f36&arch=x86_64&protocol=http&protocol=http&countme=3 [Could not resolve host: HTTPS]
  - Curl error (6): Couldn't resolve host name for http://HTTPS///mirrors.fedoraproject.org/metalink?repo=updates-released-f36&arch=x86_64&protocol=http&protocol=http [Could not resolve host: HTTPS]
Error: Failed to download metadata for repo 'updates': Cannot prepare internal mirrorlist: Curl error (6): Couldn't resolve host name for http://HTTPS///mirrors.fedoraproject.org/metalink?repo=updates-released-f36&arch=x86_64&protocol=http&protocol=http [Could not resolve host: HTTPS]
[user@dom0 ~]$ qvm-shutdown vault
[user@dom0 ~]$ qvm-service vault updates-proxy-setup -D
[user@dom0 ~]$ qvm-service vault
[user@dom0 ~]$ 
[user@dom0 ~]$ qvm-run --pass-io --no-gui vault "sudo dnf update"
Fedora 36 openh264 (From Cisco) - x86_64        0.0  B/s |   0  B     00:00    
Errors during downloading metadata for repository 'fedora-cisco-openh264':
  - Curl error (6): Couldn't resolve host name for http://HTTPS///mirrors.fedoraproject.org/metalink?repo=fedora-cisco-openh264-36&arch=x86_64&protocol=http&protocol=http [Could not resolve host: HTTPS]
Error: Failed to download metadata for repo 'fedora-cisco-openh264': Cannot prepare internal mirrorlist: Curl error (6): Couldn't resolve host name for http://HTTPS///mirrors.fedoraproject.org/metalink?repo=fedora-cisco-openh264-36&arch=x86_64&protocol=http&protocol=http [Could not resolve host: HTTPS]
Fedora 36 - x86_64 - Updates                    0.0  B/s |   0  B     00:00    
Errors during downloading metadata for repository 'updates':
  - Curl error (6): Couldn't resolve host name for http://HTTPS///mirrors.fedoraproject.org/metalink?repo=updates-released-f36&arch=x86_64&protocol=http&protocol=http [Could not resolve host: HTTPS]
  - Curl error (6): Couldn't resolve host name for http://HTTPS///mirrors.fedoraproject.org/metalink?repo=updates-released-f36&arch=x86_64&protocol=http&protocol=http&countme=3 [Could not resolve host: HTTPS]
Error: Failed to download metadata for repo 'updates': Cannot prepare internal mirrorlist: Curl error (6): Couldn't resolve host name for http://HTTPS///mirrors.fedoraproject.org/metalink?repo=updates-released-f36&arch=x86_64&protocol=http&protocol=http [Could not resolve host: HTTPS]

All good. But if vault was an App qube of a Template with no other app qube getting online to hae a successful update check reporting updates to dom0, that Template would never get update notifications pushed to dom0

Behavior of default services is:

To have updates check timer enabled as explained under prior solution post at Can somebody clarifies how qubes-update-check service works and how dom0 gets update notifications for TemplateVMs that are never powered on? - #3 by ludovic
To not have the forwarder socket enabled under app qube by default. Therefore, attempting to access updates repositories through update proxy fails even if tried by qubes update check timer.
Consequently, if there is no network enabled app qube using a specialized template, that template will never get any update notification. This is where @adw documentation update applies Can somebody clarifies how qubes-update-check service works and how dom0 gets update notifications for TemplateVMs that are never powered on? - #8 by adw

So the defaults are good and documentation update is good.
BUT: users might never manually update a Template for which no available update notification is available, leading him to use outdated sotfware without knowing it.

Documentation is good, but a warned user would be better then requiring user to read everything to be aware of everything. UX enhancement would be welcome.

My prior recommendation still applies for UX improvements of the Qubes Update Widget (@adw @ninavizz @marmarek @deeplow @Sven ) please link wherever that applies, documentation update not being enough for UX experience and deeper understanding of what truely happening when they specialize templates):

The Qubes Update widget should tell the user of the last time dom0 received an update notification for a specific template directly under Qubes Update widget
- Adding a “Last received update notification” timestamp column, stating the last time an update notification was received from dom0 would most probably be enough?
The user should be able to easily notice that other greyed templates in the Qubes Update widget have not received dom0 notifications since different dates, while updates were available for other templates.
- That should be enough to entice users into ticking the “enable updates for qubes without known available updates” which will give the real deal. If there is still no updates, nor errors given when manually attempting to update softwares in a Template: an end-user should be worried and look for EOL notices or at least minimally investigate the potential issue (unless he uses a really old template intentionally for development purposes, there should not be any reason for using deprecated and unmainted software on a daily use. Maybe that Template should even be deleted. My point is that it should be as visible as possible that unmaintained Templates are being installed, and potentially used by and only by offline qubes which don’t warn users of available updates since offline quebes don’t notify dom0 of available updates.)
- Adding an additional column “Last update check” should permit the user to notice EOL Templates easily since the timestamp for “Last received update notification” would still be old still old even if “Last update check” is recent.

Let’s remember that @unman’s shaker’s cacher is a life changer here and would deserve attention to improve UX of software installation, update time/bandwidth needed when using Qubes and ease UX installation experience combined with extrepo-date upstream’s project involvement in the goal to ease GPG public key installation and repository information addition in user’s Templates.