Dom0 update failed on v4.1.2 R4.1

Hi all,
i had a notice to update dom0 and other VM. All updates where ok but only dom0 failed. I tried it via GUI update symbol, via command line with sudo qubesctl --show-output state.sls update.qubes-dom0 and
qubes-dom0-update, without success.
A try via the update button in Qube Manager shows me the following output in terminal: (abbreviated for better understanding):

Total download size: 110 M
DNF will only download packages for the transaction.
Downloading Packages:
(1/14): kernel-6.1.35-1.qubes.fc32.x86_64.rpm   1.6 MB/s | 11 MB    00:06
....
....
[MIRROR] kernel-modules-6.1.35-1.qubes.fc32.x86_64.rpm: Curl error (23): Failed
writing received data to disk/application for https://mirrors.edge.kernel.org/qu
bes/repo/yum/r4.1/current/dom0/fc32/rpm/kernel-modules-6.1.35-1.qubes.fc32.x86_6
4.rpm [Failure writing output to destination]
[FAILED] kernel-modules-6.1.35-1.qubes.fc32.x86_64.rpm: Curl error (23): Failed
writing received data to disk/application for https://mirrors.edge.kernel.org/qu
bes/repo/yum/r4.1/current/dom0/fc32/rpm/kernel-modules-6.1.35-1.qubes.fc32.x86_6
4.rpm [Failure writing output to destination]
(6-7/14): qubes-core 89% [=================-  ] 12 MB/s | 98 MB    00:00 ETA
The downloaded packages were saved in cache until the next successful transactio
n.
You can remove cached packages by executing 'dnf clean packages'.
Error: Error downloading packages:
  Curl error (23): ............ same as above
Fetching updates failed with code 1; press Enter to exit

has anyone an idea whats going wrong here?

Do you have free disk space on dom0?

df -h executed in dom0 terminal gives me:

Filesystem                   Size  Used Avail Use% Mounted on
devtmpfs                     1.9G     0  1.9G   0% /dev
tmpfs                        1.9G     0  1.9G   0% /dev/shm
tmpfs                        1.9G  2.7M  1.9G   1% /run
/dev/mapper/qubes_dom0-root   20G  6.3G   13G  34% /
tmpfs                        1.9G  8.0K  1.9G   1% /tmp
xenstore                     1.9G  584K  1.9G   1% /var/lib/xenstored
/dev/sdd2                    974M  157M  750M  18% /boot
/dev/sdd1                    599M  104M  496M  18% /boot/efi
tmpfs                        389M   28K  389M   1% /run/user/1000

I’m having dom0 updates issues as well but mine, until today, was failing on firewall-sys.

Today’s new feature has dom0 just spinning without failing.

Yesterday after updating via Qube Manager the GUI symbol ‘Updates are vailable’ disappeared (but the update wasn’t successful). Today the notification about new updates reappeared, here’s the output of the updater:

Updating dom0

Error on updating dom0: Command '['sudo', 'qubesctl', '--dom0-only', '--no-color', 'pkg.upgrade', 'refresh=True']' returned non-zero exit status 1.
[ERROR   ] Command 'systemd-run' failed with return code: 1
[ERROR   ] stderr: Running scope as unit: run-ra2931055434341ac9f86b71e8e0a8738.scope
Using sys-firewall as UpdateVM to download updates for Dom0; this may take some time...
Running '/usr/lib/qubes/qubes-download-dom0-updates.sh --doit --nogui '--exclude=qubes-template-*' '--quiet' '-y' '--clean' '--action=upgrade'' on sys-firewall
sys-firewall: command failed with code: 1
[ERROR   ] retcode: 1
Error running 'pkg.upgrade': Problem encountered upgrading packages. Additional info follows:

changes:
    ----------
result:
    ----------
    pid:
        11311
    retcode:
        1
    stderr:
        Running scope as unit: run-ra2931055434341ac9f86b71e8e0a8738.scope
        Using sys-firewall as UpdateVM to download updates for Dom0; this may take some time...
        Running '/usr/lib/qubes/qubes-download-dom0-updates.sh --doit --nogui '--exclude=qubes-template-*' '--quiet' '-y' '--clean' '--action=upgrade'' on sys-firewall
        sys-firewall: command failed with code: 1
    stdout:

by the way im really confused now because i just realized that my system runs Debian based dom0. So why is there an error in the first output (by updating via Qube Manager) that refers to .rpm packages???

It really isn’t, and I don’t know what would bring you to that
conclusion.
You are confused.

Stop using the GUI - it’s being rewritten for 4.2 precisely because the
feedback and trouble shooting is difficult using salt.
For troubleshooting purposes, it’s best to run the update from the
command line. (This is not a general recommendation - but in these cases
it is the best thing to do.)
Open a terminal in dom0.
Run sudo qubes-dom0-update
A window should open in your updateVM, and you will be able to watch the
progress of the update there, and then see the output in dom0.
The salt output clearly indicates that the problem is with sys-firewall.
What template are you using there? Have you tried shutting down
sys-firewall, changing the template that it uses, and attempting an
update ? Sometimes this can be a quick fix (although it does not
diagnose or solve the underlying problem.)

I never presume to speak for the Qubes team. When I comment in the Forum or in the mailing lists I speak for myself.

It really isn’t, and I don’t know what would bring you to that
conclusion.
You are confused.

Yes, you are right. I missinterpreted the first line of the Qube Manager:
“Name dom0” and “Default DispVM = default(debian-11-dvm)”. I am new with Qubes OS, my mistake.

I followed the suggestion, changed the template and tried an update via
command line again.
The only output i got was about missing memory on disk (write error 0xabcdefg, no space left on disk.)

As i already wrote above gives the sudo df -h and the sudo lvs commands no indication of missing disk space.

I cleaned the cache with sudo dnf clean all and tried it again to perform an update, but the error still persists.

Btw, the disk i use is an brand new SSD, so broken areas on disk are no issue in this case.

Tried to use other qubes as the default update qube. No matter which qube i use, it doesn’t work with any other qube.

Solved it.

The reason why no update on dom0 was possible is too low memory in the update qube.

On my system the update qube is sys-firewall. I could’n increase the memory from sys-firewall because the option in qube settings was greyed out.
sys-firewall depends on a template debian-11-dvm which is a template for disposables, this in turn depends on the basic debian-11 template.
At this basic underlying template it was possible to increase the system storage max size by a few gigabytes.
This did the job, the installation of dom0 update worked.

Thanks at all for replies.

1 Like

You mean you lacked free disk space in your dvm template, not system memory? Good you figured this, that didn’t seem straightforward to debug.

You mean you lacked free disk space in your dvm template, not system memory?

Yes, what i wrote is perhaps misleading. I lacked free space in the template. The parameter i had to change is called “System storage max size”. It’s not in the dvm template but in the dvm’s underlying template “debian-11”. To increase memory in all derived templates you have to edit the underlying template.
The parameter that had to be set is found in the basic debian-11 template:

Template: debian-11debian-11: Qube SettingsTab "Basic"
section "Disk storageSystem storage max size:

Increasing the “Private storage max size” took no effect, i had to increase the “System storage max size” from 10GB to 15GB.

This affected the debian-dvm template from which the “update qube” (in my case sys-firewall) inherits. This is my understanding of the connections, in any case it now works like this.