Template doesn’t boot after update (no space left on device)

unable to update Debian-11 template for the last 2 weeks now. I am also unable to open Run-Terminal in that template. Not sure if this could indicate a compromise. Here is the output from the failed update:

Updating debian-11apps

Error on updating debian-11apps: Command ‘[‘sudo’, ‘qubesctl’, ‘–skip-dom0’, ‘–targets=debian-11apps’, ‘–show-output’, ‘state.sls’, ‘update.qubes-vm’]’ returned non-zero exit status 20.
debian-11apps:

        ID: update
  Function: pkg.uptodate
    Result: False
   Comment: W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://deb.qubes-os.org/r4.1/vm bullseye InRelease: Splitting up /var/lib/apt/lists/deb.qubes-os.org_r4.1_vm_dists_bullseye_InRelease into data and signature failed
            W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://repo.protonvpn.com/debian stable InRelease: Splitting up /var/lib/apt/lists/repo.protonvpn.com_debian_dists_stable_InRelease into data and signature failed
            W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://deb.debian.org/debian bullseye InRelease: Splitting up /var/lib/apt/lists/deb.debian.org_debian_dists_bullseye_InRelease into data and signature failed
            E: Failed to fetch https://deb.debian.org/debian/dists/bullseye/InRelease  Splitting up /var/lib/apt/lists/deb.debian.org_debian_dists_bullseye_InRelease into data and signature failed
            E: Failed to fetch https://deb.debian.org/debian-security/dists/bullseye-security/InRelease  Error writing to file - write (28: No space left on device) [IP: 127.0.0.1 8082]
            E: Failed to fetch https://repo.protonvpn.com/debian/dists/stable/InRelease  Splitting up /var/lib/apt/lists/repo.protonvpn.com_debian_dists_stable_InRelease into data and signature failed
            E: Failed to fetch https://deb.qubes-os.org/r4.1/vm/dists/bullseye/InRelease  Splitting up /var/lib/apt/lists/deb.qubes-os.org_r4.1_vm_dists_bullseye_InRelease into data and signature failed
            E: Failed to fetch https://updates.signal.org/desktop/apt/dists/xenial/InRelease  Error writing to file - write (28: No space left on device) [IP: 127.0.0.1 8082]
            E: Some index files failed to download. They have been ignored, or old ones used instead.
            W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list.d/signal-desktop.list:1 and /etc/apt/sources.list.d/signal-xenial.list:1
            W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/signal-desktop.list:1 and /etc/apt/sources.list.d/signal-xenial.list:1
            W: Target Translations (main/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list.d/signal-desktop.list:1 and /etc/apt/sources.list.d/signal-xenial.list:1
   Started: 07:07:23.498025
  Duration: 6125.424 ms
   Changes:   

        ID: notify-updates
  Function: cmd.run
      Name: /usr/lib/qubes/upgrades-status-notify
    Result: False
   Comment: Command "/usr/lib/qubes/upgrades-status-notify" run
   Started: 07:07:29.695509
  Duration: 5572.942 ms
   Changes:   
            ----------
            pid:
                38018
            retcode:
                100
            stderr:
            stdout:

Summary for debian-11apps

Succeeded: 0 (changed=1)
Failed: 2

Total states run: 2
Total run time: 11.698 s

You can see this error if /tmp is full, or if the disk is full.
Check that.

you can then try:

apt-get clean
mv /var/lib/apt/lists /tmp
mkdir -p /var/lib/apt/lists/partial
apt-get clean
apt-get update

It’s unlikely to be a sign of compromise - it’s not uncommon.

1 Like

Yea but the terminal wont open in that template vm.

qvm-console-dispvm

Newb here - Whats the rest of the command? The template is not a disp.

I tried that and got this :

Updating debian-11

Error on updating debian-11: Command ‘[‘sudo’, ‘qubesctl’, ‘–skip-dom0’, ‘–targets=debian-11’, ‘–show-output’, ‘state.sls’, ‘update.qubes-vm’]’ returned non-zero exit status 20.
debian-11:

        ID: update
  Function: pkg.uptodate
    Result: False
   Comment: W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list.d/alexp.list:1 and /etc/apt/sources.list.d/alexp.list:2
            W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/alexp.list:1 and /etc/apt/sources.list.d/alexp.list:2
            W: Target Translations (main/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list.d/alexp.list:1 and /etc/apt/sources.list.d/alexp.list:2
            W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list.d/alexp.list:1 and /etc/apt/sources.list.d/alexp.list:3
            W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/alexp.list:1 and /etc/apt/sources.list.d/alexp.list:3
            W: Target Translations (main/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list.d/alexp.list:1 and /etc/apt/sources.list.d/alexp.list:3
            E: Failed to fetch https://dl.google.com/linux/chrome/deb/dists/stable/InRelease  Invalid response from proxy: HTTP/1.0 500 Unable to connect  Server: tinyproxy/1.10.0  Content-Type: text/html  Connection: close     [IP: 127.0.0.1 8082]
            E: Failed to fetch https://contrib.qubes-os.org/deb/r4.1/vm/dists/bullseye/InRelease  Certificate verification failed: The certificate is NOT trusted. The certificate chain uses expired certificate.  Could not handshake: Error in the certificate verification. [IP: 127.0.0.1 8082]
            E: Some index files failed to download. They have been ignored, or old ones used instead.
            W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list.d/alexp.list:1 and /etc/apt/sources.list.d/alexp.list:2
            W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/alexp.list:1 and /etc/apt/sources.list.d/alexp.list:2
            W: Target Translations (main/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list.d/alexp.list:1 and /etc/apt/sources.list.d/alexp.list:2
            W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list.d/alexp.list:1 and /etc/apt/sources.list.d/alexp.list:3
            W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/alexp.list:1 and /etc/apt/sources.list.d/alexp.list:3
            W: Target Translations (main/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list.d/alexp.list:1 and /etc/apt/sources.list.d/alexp.list:3
   Started: 08:32:05.661146
  Duration: 3030.245 ms
   Changes:   

        ID: notify-updates
  Function: cmd.run
      Name: /usr/lib/qubes/upgrades-status-notify
    Result: False
   Comment: Command "/usr/lib/qubes/upgrades-status-notify" run
   Started: 08:32:08.698999
  Duration: 2911.901 ms
   Changes:   
            ----------
            pid:
                4521
            retcode:
                100
            stderr:
            stdout:

Summary for debian-11

Succeeded: 0 (changed=1)
Failed: 2

Total states run: 2
Total run time: 5.942 s

You were having the same issue?

It’s not the same issue

If you run the command as given you will receive a helpful message that
tells you how to use it,and exactly what it does.

If you do not know what a command does, you can try to run it with a -h
or --help option - very often in Linux, and error message will tell
you how to get help.
You can also use man - like man qvm-console-dispvm - which should
provide you with a more detailed explanation of the command and
available options.

Okay.

It should, but unfortunately, in this case, it does not. Opened a bug report:

can you provide me the exact solution and commands? nothing seems to be working for me.

@NicholasTrust contacted me directly, off channel, for this issue. Replying here, but this is not a new issue. Agreeing with @unman with most probable cause being lack of space under template, where normally apt refuses to apply update if forecasted consumed space is bigger than actual available space. But that might have been gone wrong.

This issue should be renamed to “Template doesn’t boot after update” for prosperity and usefulness in the future for others.


I would advise getting safe belts first:
qvm-clone debian-11 debian-11-fixed

qvm-console-dispvm as of 09 february 2023 gives:

Usage: /usr/bin/qvm-console-dispvm [--autostart] [--] vmname
Connects to VM console throught DispVM using the qubes.ShowInTerminal RPC service.
With --autostart, start the VM first.

So if the RPC service will still be listening into broken (to be fixed) template named debian-11-fixed, it would be
qvm-console-dispvm --autostart debian-11-fixed which will open a terminal in dispvm to the broken template.

And then resume from previously given instructions.

That would be running df -h from the dispvm terminal.
On a template:
/dev/mapper/root here is the template OS layer filesystem. You want to make sure there is enough free space there and if that was the cause. You can increase that root filesystem space from Qubes Settings of that template, there named System storage max size where Private storage max is maximum size of /home (qubes persistence filesystem). Again doing changes at that layer might require a reboot reboot (df -h might infirm this) but changes need to be applied to be useful.

Once you can open a terminal normally under debian-11-fixed (which would mean you fixed the template, hence its hypothetical name), you might want to create a new qube using that template first and see if that qube boots normally. Then delete that testing qube.

Then you can use the Qubes Template Manager to have all you qubes point to now fixed debian-11-fixed template. Then Qubes manager to rename debian-11 to debian-11-backup, and debian-11-fixed to debian-11. And then qvm-remove debian-11-backup later on, requiring qvm-prefs debian-11-backup installed_by_rpm False otherwise qvm-remove debian-11-backup will fail, saying that it was installed by rpm…

As usual, you need to have all qubes depending on those templates shutdown. I use qvm-shutdown --force --all --wait from dom0 because I know what i’m doing, but you could point to individual qubes directly through: qvm-shutdown --force NameOfQubeDependingOnTemplate1 NameOfQubeDependingOnTemplate2.

Make sure everything works as expected by pointing one qube, or creating a new testing qube pointing to your repaired template first. Otherwise causing more issue then anything else.

You could also just redownload debian-11 template from the qvm-template-gui tool from dom0.

In the future, when you see that something broke, that a template is not providing OS for a qube, you can easily revert the template to its prior state. Qubes provides that safeguard for all qubes and templates. The proper way to do this preventively would be to create a new qube using that template in question. If that doesn’t boot normally, you can clone the template in its current state to a backup (qvm-clone debian-11 debian-11-backup) and then call qvm-volume revert debian-11:root and retry booting your test qube using that template.

Hope things are clearer here. Note that in case a Template or a qube is consuming space reaching the limits of defined maximum System or Private storage, QubesOS is warning the user of such case on top right corner of xfce, asking the user to take preventive action, permitting to click the warning pointing directly to that Template/qube settings to increase assigned maximum space.


@adw : qvm-volume revert should get more publicity for a while now. Is rollbacking, directly in qubes manager, a planned feature? A hook on failing to start a qube, or a failed template update should most probably guide the user into reverting to prior of such update?

I see qvm-volume revert NameOfQube:root to be really useful for templates for rollback purposes, with no bad consequcnes. If issue like this one are opened frequently (forum/github), then QubesOS has a UX problem, where this issue shows lack of knowledge to be able to resolve easily and becoming an emergency situation. It could be a “lay back and relax, Qubes OS got your back: do you want to rollback failed update?” kinda situation instead if we really wanted to, as well as a way to promote QubesOS usage for newbies/not-so-techsavvy users who fear of breaking things.

1 Like

Renamed topic to “Template doesn’t boot after update (no space left on device)”.

EDIT: Original post clearly shows in logs errors pointing to the template being filled by attempt of downloading files from update repo:

@adw : Meaning that @NicholasTrust missed the warnings related to limited space warning popup on Template startup, or missed how to resolve the situation by clicking information pop-up, and/or didn’t know how to roll back to prior state with help of qvm-volume revert, resulting in a possible unusable system if all qubes were based on a single template not booting anymore.

Sorry, that’s a bit beyond my knowledge and experience. However, I was able to find some issues that seem relevant:

If one of these covers what you’re describing, feel free to comment there. If not, please feel free to open a new issue.

I have increased Storage. The update worked but now i am unable to reduce storage size back down to where is as before.

Not sure why you would do that but feel free to resume using Qubes official docs Resize disk image | Qubes OS

There is no need to reduce the storage back down.
In the default Qubes does not allocate more storage than you are
actually using. This is how you can over provision more qubes than you
have space on the drive.
But if you do want to reduce the allocated space it’s a
manual operation, covered here