I updated my Debian 10 template and it got destroyed - no boots for VMs based on it. I removed it as per documentation and tried reinstalling and it failed with errors listed below - “no space left on device”. I also tried installing other templates and got the same message.
I deleted large VMs and still get no space left message. I also deleted logs on Dom0 to no avail.
I can create new VMs so there is space, but something is wrong on Dom0.
Any advice on how to fix this would be much appreciated.
sudo qubes-dom0-update qubes-template-debian-10
Using disp-sys-firewall as UpdateVM to download updates for Dom0; this may take some time...
--> Running transaction check
---> Package qubes-template-debian-10.noarch 0:4.0.6-202101091318 will be installed
--> Finished Dependency Resolution
https://mirrors.phx.ms/qubes/repo/yum/r4.0/templates-itl/rpm/qubes-template-debian-10-4.0.6-202101091318.noarch.rpm: [Errno 14] curl#6 - "Could not resolve host: mirrors.phx.ms"
Trying other mirror.
https://mirrors.dotsrc.org/qubes/repo/yum/r4.0/templates-itl/rpm/qubes-template-debian-10-4.0.6-202101091318.noarch.rpm: [Errno 16] [Errno 28] No space left on device
Trying other mirror.
https://mirrors.ukfast.co.uk/sites/qubes-os.org/repo/yum/r4.0/templates-itl/rpm/qubes-template-debian-10-4.0.6-202101091318.noarch.rpm: [Errno 16] [Errno 28] No space left on device
Trying other mirror.
https://mirrors.edge.kernel.org/qubes/repo/yum/r4.0/templates-itl/rpm/qubes-template-debian-10-4.0.6-202101091318.noarch.rpm: [Errno 16] [Errno 28] No space left on device
Trying other mirror.
https://mirrors.gigenet.com/qubes/repo/yum/r4.0/templates-itl/rpm/qubes-template-debian-10-4.0.6-202101091318.noarch.rpm: [Errno 16] [Errno 28] No space left on device
Trying other mirror.
https://yum.qubes-os.org/r4.0/templates-itl/rpm/qubes-template-debian-10-4.0.6-202101091318.noarch.rpm: [Errno 16] [Errno 28] No space left on device
Trying other mirror.
https://mirror.linux.pizza/qubes-os.org/repo/yum/r4.0/templates-itl/rpm/qubes-template-debian-10-4.0.6-202101091318.noarch.rpm: [Errno 14] HTTPS Error 404 - Not Found
Trying other mirror.
https://quantum-mirror.hu/mirrors/pub/qubes/repo/yum/r4.0/templates-itl/rpm/qubes-template-debian-10-4.0.6-202101091318.noarch.rpm: [Errno 16] [Errno 28] No space left on device
Trying other mirror.
https://ftp.halifax.rwth-aachen.de/qubes/repo/yum/r4.0/templates-itl/rpm/qubes-template-debian-10-4.0.6-202101091318.noarch.rpm: [Errno 16] [Errno 28] No space left on device
Trying other mirror.
https://ftp.icm.edu.pl/pub/os/qubes/repo/yum/r4.0/templates-itl/rpm/qubes-template-debian-10-4.0.6-202101091318.noarch.rpm: [Errno 16] [Errno 28] No space left on device
Trying other mirror.
https://mirror.hackingand.coffee/qubes/repo/yum/r4.0/templates-itl/rpm/qubes-template-debian-10-4.0.6-202101091318.noarch.rpm: [Errno 16] [Errno 28] No space left on device
Trying other mirror.
https://mirrors.dgplug.org/qubes/repo/yum/r4.0/templates-itl/rpm/qubes-template-debian-10-4.0.6-202101091318.noarch.rpm: [Errno 14] curl#60 - "server certificate verification failed. CAfile: none CRLfile: none"
Trying other mirror.
https://ftp.cc.uoc.gr/mirrors/linux/qubes/repo/yum/r4.0/templates-itl/rpm/qubes-template-debian-10-4.0.6-202101091318.noarch.rpm: [Errno 16] [Errno 28] No space left on device
Trying other mirror.
https://ftp.acc.umu.se/mirror/qubes-os.org/repo/yum/r4.0/templates-itl/rpm/qubes-template-debian-10-4.0.6-202101091318.noarch.rpm: [Errno 16] [Errno 28] No space left on device
Trying other mirror.
Could not download/verify pkg qubes-template-debian-10-4.0.6-202101091318.noarch: failure: rpm/qubes-template-debian-10-4.0.6-202101091318.noarch.rpm from qubes-templates-itl: [Errno 256] No more mirrors to try.
The way installing and updating packages in dom0 works in Qubes is that packages first get downloaded to an intermediate qube (by default sys-firewall) and then they are transferred to dom0, which is offline. So most likely you are out space in sys-firewall. Maybe if you restart it, the internal cache will get cleared. If this doesn’t work, try increasing “Private storage max. size” in “Settings” for sys-firewall.
Sorry about that. You can format the post with markdown. I’ve fixed it for you. So if you format the terminal out with
```
some terminal output
https://mirrors.edge.kernel.org/qubes/repo/yum/r4.0/templates-itl/rpm/qubes-template-debian-10-4.0.6-202101091318.noarch.rpm: [Errno 16] [Errno 28] No space left on device
Trying other mirror.
```
It would look like
some terminal output
https://mirrors.edge.kernel.org/qubes/repo/yum/r4.0/templates-itl/rpm/qubes-template-debian-10-4.0.6-202101091318.noarch.rpm: [Errno 16] [Errno 28] No space left on device
Trying other mirror.
I tried the solution but it still does not work. I increased the firewall private storage significantly and still get the same errors even after a restart. Please uncheck previous answer as a solution admins.
If I keep running the command it gives me some different errors which would seem to indicate a possibly a space issue on Dom0 even though I removed quite large VMs - it appears that maybe possibly Dom0’s allocated space is an issue?
Using disp-sys-firewall as UpdateVM to download updates for Dom0; this may take some time…
tar: var/lib/rpm/__db.002: Wrote only 1024 of 10240 bytes
tar: var/lib/rpm/__db.003: Cannot write: No space left on device
tar: var/lib/rpm/__db.001: Cannot write: No space left on device
tar: etc/yum.repos.d: Cannot mkdir: No space left on device
tar: etc/yum.repos.d: Cannot mkdir: No space left on device
tar: etc/yum.repos.d/fedora-updates.repo: Cannot open: No such file or directory
tar: etc/yum.repos.d: Cannot mkdir: No space left on device
tar: etc/yum.repos.d/qubes-dom0.repo: Cannot open: No such file or directory
tar: etc/yum.repos.d: Cannot mkdir: No space left on device
tar: etc/yum.repos.d/qubes-templates.repo: Cannot open: No such file or directory
tar: etc/yum.repos.d: Cannot mkdir: No space left on device
tar: etc/yum.repos.d/fedora.repo: Cannot open: No such file or directory
tar: etc/yum.conf: Cannot write: No space left on device
tar: Exiting with failure status due to previous errors
cp: error writing ‘/var/lib/qubes/dom0-updates/home/user/.rpmdb/Packages’: No space left on device
cp: error writing ‘/var/lib/qubes/dom0-updates/home/user/.rpmdb/Basenames’: No space left on device
cp: error writing ‘/var/lib/qubes/dom0-updates/home/user/.rpmdb/Group’: No space left on device
cp: error writing ‘/var/lib/qubes/dom0-updates/home/user/.rpmdb/Providename’: No space left on device
cp: error writing ‘/var/lib/qubes/dom0-updates/home/user/.rpmdb/Conflictname’: No space left on device
cp: error writing ‘/var/lib/qubes/dom0-updates/home/user/.rpmdb/Triggername’: No space left on device
cp: error writing ‘/var/lib/qubes/dom0-updates/home/user/.rpmdb/Name’: No space left on device
cp: error writing ‘/var/lib/qubes/dom0-updates/home/user/.rpmdb/Suggestname’: No space left on device
cp: error writing ‘/var/lib/qubes/dom0-updates/home/user/.rpmdb/Requirename’: No space left on device
cp: error writing ‘/var/lib/qubes/dom0-updates/home/user/.rpmdb/Sha1header’: No space left on device
cp: error writing ‘/var/lib/qubes/dom0-updates/home/user/.rpmdb/Supplementname’: No space left on device
cp: error writing ‘/var/lib/qubes/dom0-updates/home/user/.rpmdb/Installtid’: No space left on device
cp: error writing ‘/var/lib/qubes/dom0-updates/home/user/.rpmdb/Dirnames’: No space left on device
cp: error writing ‘/var/lib/qubes/dom0-updates/home/user/.rpmdb/Enhancename’: No space left on device
cp: error writing ‘/var/lib/qubes/dom0-updates/home/user/.rpmdb/Sigmd5’: No space left on device
cp: error writing ‘/var/lib/qubes/dom0-updates/home/user/.rpmdb/Obsoletename’: No space left on device
cp: error writing ‘/var/lib/qubes/dom0-updates/home/user/.rpmdb/Transfiletriggername’: No space left on device
cp: error writing ‘/var/lib/qubes/dom0-updates/home/user/.rpmdb/__db.002’: No space left on device
error: rpmdb: BDB0137 write: 0x582b69eb0630, 4096: No space left on device
error: cannot open Basenames index using db5 - No space left on device (28)
error: rpmdb: BDB0137 write: 0x582b69eb1710, 4096: No space left on device
error: cannot open Group index using db5 - No space left on device (28)
error: rpmdb: BDB0137 write: 0x582b69eb1710, 4096: No space left on device
error: cannot open Requirename index using db5 - No space left on device (28)
error: rpmdb: BDB0137 write: 0x582b69eb1710, 4096: No space left on device
error: cannot open Providename index using db5 - No space left on device (28)
error: rpmdb: BDB0137 write: 0x582b69eb1710, 4096: No space left on device
error: cannot open Conflictname index using db5 - No space left on device (28)
error: rpmdb: BDB0137 write: 0x582b69eb1710, 4096: No space left on device
error: cannot open Obsoletename index using db5 - No space left on device (28)
error: rpmdb: BDB0137 write: 0x582b69eb1cd0, 4096: No space left on device
error: cannot open Triggername index using db5 - No space left on device (28)
error: rpmdb: BDB0137 write: 0x582b69eb1cd0, 4096: No space left on device
error: cannot open Dirnames index using db5 - No space left on device (28)
error: rpmdb: BDB0137 write: 0x582b69eb1d70, 4096: No space left on device
error: cannot open Installtid index using db5 - No space left on device (28)
error: rpmdb: BDB0137 write: 0x582b69eb17b0, 4096: No space left on device
error: cannot open Sigmd5 index using db5 - No space left on device (28)
error: rpmdb: BDB0137 write: 0x582b69eb17b0, 4096: No space left on device
error: cannot open Sha1header index using db5 - No space left on device (28)
error: rpmdb: BDB0137 write: 0x582b69eb17b0, 4096: No space left on device
error: cannot open Filetriggername index using db5 - No space left on device (28)
error: rpmdb: BDB0137 write: 0x582b69eb17b0, 4096: No space left on device
error: cannot open Transfiletriggername index using db5 - No space left on device (28)
error: rpmdb: BDB0137 write: 0x582b69eb3ee0, 4096: No space left on device
error: cannot open Recommendname index using db5 - No space left on device (28)
error: rpmdb: BDB0137 write: 0x582b69eb3ee0, 4096: No space left on device
error: cannot open Suggestname index using db5 - No space left on device (28)
error: rpmdb: BDB0137 write: 0x582b69eb3ee0, 4096: No space left on device
error: cannot open Supplementname index using db5 - No space left on device (28)
error: rpmdb: BDB0137 write: 0x582b69eb3ee0, 4096: No space left on device
error: cannot open Enhancename index using db5 - No space left on device (28)
error: db5 error(11) from dbenv->open: Resource temporarily unavailable
error: cannot open Packages index using db5 - Resource temporarily unavailable (11)
error: cannot open Packages database in /var/lib/qubes/dom0-updates/home/user/.rpmdb
CRITICAL:yum.verbose.cli.yumdownloader:Yum Error: Error: rpmdb open failed
So maybe you did run out of space. If you click on the “Qubes Disk Space Monitor” in the system tray, it should tell you how much space you have left. I have no other ideas. Sorry!
Qubes Disk Space Monitor says there is more than 40% left, at least 100GB. Cant be actual disk space issue as I have removed over 30GB of VMs and cleared dom0 logs.
Try increasing the "System storage max. size" in the TemplateVM(for disp-sys-firewall) settings from the default value of 10240MiB to 16384MiB or higher and then reboot disp-sys-firewall.
If that doesn’t work, would you mind sharing the output of the following command in disp-sys-firewall terminal?