/dev/mapper/qubes_dom0-root 100% full?

I sat down in front of my computer today and I noticed I have an error, I clicked on it and it says that I have 0% space left on my home folder? the disk notification says I am using 99.9% of my varlibsqubes data volume but I honestly am not entirely sure what that means? and definitely not sure why.

I did a df and indeed i seem to be out of space but I have no idea why and can’t seem to find what is taking all the space not to mention how to stop it. Somehow my computer is still working but in reading i only found information on how to clear some space but not diagnosing why/how? I am hesitant to turn off my computer as from reading it sounds like I might not be able to get back into the gui?

Any ideas on how i can diagnose what is taking up all the space?

Hi Stumpi,

Open up a terminal in Dom0 and run as root ( sudo -i) from top level directory:

du -cks * | sort -rn | head -11

This was my output, sorted from largest to smallest

[klaus@dom0 ~]$ sudo -i
[root@dom0 ~]# cd /
[root@dom0 /]# du -cks * | sort -rn | head -11
du: cannot read directory 'proc/5617/task/5617/net': Invalid argument
du: cannot read directory 'proc/5617/net': Invalid argument
du: cannot access 'proc/10930/task/10930/fd/4': No such file or directory
du: cannot access 'proc/10930/task/10930/fdinfo/4': No such file or directory
du: cannot access 'proc/10930/fd/3': No such file or directory
du: cannot access 'proc/10930/fdinfo/3': No such file or directory
11415192	total
5057860	usr
3468304	home
2558240	var
270792	boot
55696	etc
2452	run
976	srv
688	root
152	tmp
16	lost+found
[root@dom0 /]#

From there you can cd into the directory you want to know more, and run the command again, drilling down to the greatest size files in the path. Let me know if you have any quesitons, and happy hunting!

1 Like

This was educational, it uses a lot of things I didn’t realize existed. (Such as du not showing everything all at once!)

A couple of observations/questions.

I’ve never used -i on sudo; when I try it (sudo -i du -cks * | sort -rn), I get nothing but error messages; it cannot access any of the root level directories. If I combine -i with -u then it’s able to read /bin but nothing else. I’m inferring there’s something else “obvious” I should be doing (so obvious you didn’t specify it).

EDIT: I think I may have just found the “obvious” thing. sudo -i should be run on a command line by itself. Then cd to / and then issue the du command.

I had vastly better results with just plain “sudo” but there are eight directories it cannot read even as sudo and it spits out ugly error messages. (The directories all start with proc.) Fortunately you can read around this.

More concretely useful to the original poster, perhaps:
My home directory is much, much smaller than /usr (188,640 versus 7,035,788). Honestly if I got the numbers you showed I’d start drilling down in /home. Another place to start is /var; there might be some gargantuan but useless log file(s) in there somewhere.

As to the ‘sudo -i’ command, the -i acquires the root users environment settings when becoming root. I used that by itself to become root and executed all commands afterwards from the top level directory.

I guess I should have been more clear by posting the entire session, so you could see how I ran commands.

I literally was writing my edit (where I figured this out) as you posted.

The reason I figured it out was I tried to close the terminal with control-d. It logged me out of a session but stayed open, it took another control-d to close the terminal. I then realized in my fumblings I had managed to create a shell within a shell, and poking around with -i and no command made me realize I must have done that by mistake. I then tried doing it deliberately, and the du command worked as advertised once I cd’d to /. (oh, and do NOT use -u )

If I might make a suggestion: you could edit your post to show things.

Done and done! Post now correctly shows all commands used :wink: Thanks SteveC for the suggestion…

Thank you both!
Well it was some ridiculously large logs but before I delete them I wanted to check why they might have been generated and if I should (somehow) post them?
they are /var/log/xen/console/guest-sys-net.log* (3 of them all over 1g, 9+g total).
Should it be ok to just delete them or should the fact that they were generated be of concern?

For the benefit of other: The issue is here:

/var/log/xen/console/guest-sys-usb.log

Booted from a Fedora 39 live USB, luksOpen’ed the qubes_dom0-root, mounted it then dumped the last 50 lines in a tmp file then overwrote the original.

My varlibqubes disk space usage after a reboot went from 100% to 37%

There needs to be a cron job or similar to do this automatically. An unlimited size for the sys-usb log is not desirable.