Qubes won’t start - not enough disk space

I have a made an error with my backup; I backed up to file space in a Qube which didn’t complete because it ran out of space. I assumed the file would not save if the backup didn’t complete but unfortunately it did; I rebooted and now the OS won’t start due to not having enough space. How do I fix this?

Boot in a live usb (even the Qubes OS installer, enter TTY with CTRL+ALT+F2) and deal with the qube in question from there.

# If your QubesOS installation is encrypted:
$ cryptsetup open /dev/<device> luks-qubes
# Find the name of the qube of interest and mount it
$ lsblk
# Grep can help you locate the exact qube
$ lsblk | grep <my-qube>
$ mount /dev/mapper/qubes_dom0-vm--<my-qube>--private /mnt
# Do what you need to do then unmount
$ umount /mnt
$ vgchange -a n
$ cryptsetup close luks-qubes

(you may need to run the above as root)

Done.

1 Like

I’m new to Qubes; just some more info that might help, I have a Lenovo T430, during the boot process I press default boot and enter the password for my SSD and that’s as far as it gets before error messages.

If I understand what your saying I should download qubes onto a usb and boot from there and mount the qube in question? What is a TTY?

Thank you

Could you report what errors you see?

You can use the usb that you used to install Qubes. Or any other live distro for that matter, I suggested the Qubes installer because it has no connectivity.

Correct. Once it’s mounted you can remove the file that causing you troubles (thus freeing up space).

In live distros you have the option to use a terminal, while in the qubes installer you do not. That’s when TTY comes in. In easy terms it basically behaves like a full screen terminal.
You can read more about it here: What is tty within Linux? - Unix & Linux Stack Exchange


Please see attached for errors. Also, I bought the T430 with Qubes pre-installed with a nitrokey, does this change your solution?

No, it just means that you may want to create a usb qubes installer. Instructions here: Installation guide | Qubes OS
But before that, try what I suggest below.

Regarding the second image, with the [Dom0] Backup error:

  1. What happens if you press OK?
  2. If you press CTRL + ALT + F2 do you get into TTY?
  3. If you do, you may be asked for your login credentials. Log in and run cat ~/.xsession-errors, then report the errors here.
  4. Finally, if you do get to TTY, you may be able to access the qube in which the backup was saved: lsblk | grep <my-qube>

I’ve probably confused you with the second image; this was the error I got when the back up failed. I hit ok and should’ve deleted the file it created but instead rebooted which results in there being not enough space and the error message from the first image.

Alright, no problem. Then create a USB installer (like I said, can be Qubes, can be anything else) and proceed with the solution proposed in my first comment.

Thank you @BEBF738VD. Can anything be done from the recovery shell as I can access this?

Maybe, I don’t know what commands are available there. You can try though.

Log in with your root password and check if cryptsetup and mount work. If they do, you can fix it from the recovery shell.

Hello @BEBF738VD I’m going through your suggested above; I have a nitrokey which works in conjunction with Heads (osreach.net). The ctrl alt f2 doesn’t work but within the Heads UI I’m able to select boot from USB. Once I select this I have a number of options:
Troubleshoot - verbose boot and install
Test media and install ( isolinux or pxeboot )
Run a memory test
Rescue qubes OS - ( isolinux or pxeboot )
Install qubes - ( isolinux or pxeboot )
Install Qubes in basic graphics

Are you still able to advise?

Thank you

Where did you run it?

This. Then, when you reach the installation welcome screen (where you choose the language), hit CTRL ALT F2 and proceed with what I suggested above.

I ran it as soon as I started the machine.

Thank you I’ll give it a go.

Hello @BEBF738VD, The boot doesn’t work when selecting the basic graphics option. It just sits on starting new kernel and doesn’t do anything I can’t hear or see anything happening? I’ve downloaded onto 2 separate USBs with the same outcome, any suggestions?

Thanks

Try the other options. Then re-try them with secure boot disabled.

And see if you find anything relevant in the docs: Installation troubleshooting | Qubes OS

Your goal is to get to a terminal on the machine, it doesn’t matter how this is achieved. It can even be another distro.

Hello @BEBF738VD - it looks like I didn’t save to qube but to dev/sda as the volume on this is zero. I’ve tried looking here and on the mounted on location and can’t find anything that resembles a back up file what extension am I looking for and is it safe to delete in these locations considering I have a nitrokey that relies on files in certain folders not to be changed?

Is /dev/sda your QubesOS drive or an external flash drive? Because in the latter case, it shouldn’t be giving you problems.

What do you mean?

Which location exactly?

Which locations?

And I’m not familiar with the nitrokey, I believe there’s a doc about it somewhere.

Hi @BEBF738VD; It’s on the Qubes OS drive. Please have a look at the pictures attached hopefully they’ll make what I’m trying to say clearer.


Thank you

Can you confirm backup file extension so I can narrow my search?

Mount the root fs:

mount -r /dev/qubes_dom0/root /mnt

then show me the output of:

cat /mnt/etc/fstab

Feel free to truncate UUIDs.

Also, a tip: you can order disks by size with:

lsblk -o NAME,SIZE -x SIZE

search for qubes-backup-2022 (it’s not a file extension but the beginning of the file name)

Thank you @BEBF738VD I think the picture gives you everything you asked for? At the bottom of the picture you’ll see the qubes file; I take it this is the backup file?