Laptop won't boot after dom0 update got interrupted, stuck in emergency mode, fsck shows I/O error

User Support
1

Hey everyone, hoping for some help. My Qubes laptop stopped booting after a dom0 update got cut off partway through (a low-memory popup appeared and the update just stopped). Since then it won’t boot normally anymore.

System: Qubes R4.3, kernel 7.1.6

What happens on boot

Every time I boot, it drops into a black screen saying:

Entering emergency mode. Exit the shell to continue.
Type "journalctl" to view system logs.

What the logs show

journalctl points to the root filesystem failing its check:

systemd-fsck-root.service: Main process exited, code=exited, status=1/FAILURE
Failed to start systemd-fsck-root.service - File System Check on /dev/mapper/qubes_dom0-root.
Dependency failed for /sysroot.

/dev/mapper/qubes_dom0-root: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY. (i.e., without -a or -p options)

What I tried

I ran this from the emergency shell:

fsck -y /dev/mapper/qubes_dom0-root

It found and fixed a bunch of errors (inode reference counts, free block counts, etc.), but right at the end it printed:

Error writing file system info: Input/output error
/dev/mapper/qubes_dom0-root: ***** FILE SYSTEM WAS MODIFIED *****

I rebooted anyway, but it’s still stuck at the same emergency mode screen.

My main worry

That “Input/output error” at the end, does that usually mean the drive itself is failing (bad sectors, dying SSD/NVMe), or can it just be leftover damage from the update getting killed mid-write? I don’t want to keep running fsck if the disk is actually dying.

What I’m hoping someone can help with

  1. Should I check the drive’s health (SMART data) before trying fsck again?

  2. What’s the best way to check that from the Qubes/dracut emergency shell? For example, is this the right command:

    smartctl -a /dev/nvme0

  3. If the drive turns out to be fine, what’s the correct next step to finish the repair?

  4. Would it be smarter to boot from a Qubes installer USB or a Linux live USB to repair the disk while it’s not in use?

I can only post screenshots (no copy-paste from that machine), but I’m happy to run any command and share a clear photo of the output.

Thanks in advance, trying not to make things worse before I understand what’s actually going on.

2

Probably the latter - LVM running out of space for the pool hosting the dom0 filesystem, making that pool read-only as a response, and now something like fsck that tries to write to it gets an I/O error.

I hope somebody else can help you with that, as I’m not experienced with LVM recovery.

3

Oh alright thank you

4

I am very ignorant about LVM recovery, but I have this bookmark from redhat - some of the things cannot be un-done, so I woul recommend to make a backup of everything before trying to repair…

…but the “gathering information” parts may help. I saved it a little time ago, but I cannot guarantee it was not made by AI:

Edit: Sorry - it was for @Martin32

5

Fixed it, turned out to be exactly what a couple of people suspected: the LVM thin pool ran out of space and locked itself read-only, which is why fsck kept throwing “Input/output error” instead of actually repairing anything.

For anyone else who hits this, here’s what worked:

1. Boot into the dracut emergency shell you’re already dropped into (no separate rescue USB needed).

2. Activate the volume group (note: this minimal shell doesn’t have separate vgchange/lvs/etc. commands, everything has to go through the combined lvm binary):

lvm vgchange -ay

3. Check actual free space on the drive:

lvm vgs qubes_dom0

I had ~42GB sitting unallocated (VFree) that wasn’t assigned to either pool.

4. Important: there are two separate pools — vm-pool (your qubes) and root-pool (dom0’s own root filesystem). Figure out which one is actually backing the volume that’s failing. My root filesystem lives on root-pool, so I had to fix that one specifically, not just vm-pool.

5. Deactivate the whole volume group first (deactivating just the pool fails while thin volumes on top of it are still active):

lvm vgchange -an qubes_dom0

6. Extend the pool that’s actually short on space:

lvm lvextend -L +5G qubes_dom0/root-pool
lvm lvextend -L +5G qubes_dom0/vm-pool

7. Repair it:

lvm lvconvert --repair qubes_dom0/root-pool
lvm lvconvert --repair qubes_dom0/vm-pool

8. Reactivate everything:

lvm lvchange -ay qubes_dom0/root-pool
lvm lvchange -ay qubes_dom0/vm-pool
lvm lvchange -ay qubes_dom0/root

9. Then finally run fsck:

fsck -y /dev/mapper/qubes_dom0-root

It fixed a bunch of inode ref-count errors and finished clean with no I/O error this time. Rebooted normally and everything came back up.

If you hit this: check lvm lvs -a -o+lv_when_full,data_percent,metadata_percent qubes_dom0 first to see which specific pool is actually full before touching anything — don’t assume it’s vm-pool just because that’s the “main” one, since dom0’s own root filesystem has its own separate pool too.

Also worth doing afterward: set thin_pool_autoextend_threshold in /etc/lvm/lvm.conf so pools grow automatically before hitting 100% again.

6

Thank you very much for the help

7

I think I did not do much…

Your explanation of your fix seems very clear - thank you for writing it all. I am keeping a bookmark for it!

It seems annoying that a disk-full problem can look like a disk-fail problem. Maybe there was some clue in dom0 journal?

8

How should we interpret Martin’s statement again?

When exactly does a hard drive like this become unresponsive, even when there is still enough free space available?

9

Yeah there are some messages in the kernel log when this happens:

But if they’re about the root pool (not the VM pool), they won’t survive a restart.