vm-pool thin metadata stuck in repair loop (transaction id mismatch) — all qubes fail to start

User Support
1

vm-pool thin metadata stuck in repair loop (transaction id mismatch) — all qubes fail to start

Qubes version: R4.x
Hardware: Lenovo ThinkPad X230
Storage: Two physical volumes in qubes_dom0 VG — internal Samsung SSD 870 EVO (sda3) and Kingston SKC600 (sdb1), both LUKS-encrypted

Summary

After installing pending dom0/template updates, on next boot every qube fails to start (sys-net, sys-firewall, sys-usb, work, vault, etc.) with:

Qube <name> has failed to start: Check of pool qubes_dom0/vm-pool failed (status:64). Manual repair required!

I suspect this followed an unclean shutdown around the time of the update (system may have frozen or been force-powered-off).

What I’ve tried

  1. Found and removed several stale -snap volumes left over from an unclean shutdown (vm-sys-net-root-snap etc.) — these were re-created/blocked initially but eventually cleared.
  2. Ran sudo lvconvert --repair qubes_dom0/vm-pool from dom0. Result:
    Transaction id 4126 from pool "qubes_dom0/vm-pool" does not match repaired transaction id 4125 from /dev/qubes_dom0/lvol0_pmspare.
  3. Re-ran the repair multiple times (including after qvm-kill on stuck qubes) — same error every time, transaction ids unchanged (4126 vs 4125).
  4. Booted a SystemRescue (13.01) live USB, unlocked both LUKS volumes (cryptsetup luksOpen on sda3 and sdb1), ran vgchange -ay qubes_dom0 successfully, confirmed lvol0_pmspare exists (68.00m) and vm-pool_tdata/vm-pool_tmeta are present but inactive.
  5. Re-ran lvconvert --repair qubes_dom0/vm-pool from the clean SystemRescue environment (no Qubes daemons, no other processes touching the pool) — identical mismatch, same transaction ids (4126/4125).
  6. Tried lvchange --activationmode degraded -ay qubes_dom0/vm-work-private — blocked with same “Check of pool … failed (status:64)”.
  7. Tried disabling thin_check_executable in /etc/lvm/lvm.conf and via lvchange --config 'global{thin_check_executable=""}' — still blocked.

Current state

  • vm-pool, vm-pool_tdata, vm-pool_tmeta all inactive, not corrupted-looking at the LV level (sizes look normal: ~1.01t for tdata, 68.00m for tmeta).
  • lvol0_pmspare exists and presumably holds a validly-repaired metadata copy, but LVM refuses to commit it due to the transaction id being one behind current.
  • No vm-pool_meta0/_meta1 backup volume has appeared yet (unlike some other reports of this issue) — repair seems to validate but never complete the swap.
  • I have NOT yet tried deactivating tdata/tmeta individually and reactivating, or manually inspecting metadata with thin_dump/thin_repair, since I wanted community input before going further given this is my only copy of the data.

Question

Has anyone resolved this specific “repair validates but transaction id mismatch blocks commit” loop? I found similar reports (qubes-issues #6212, #6213, #6214) but no confirmed fix posted there. I’m prioritizing getting data off (vm-work-private, vm-vault-*, etc.) before attempting anything further. Open to trying thin_dump/thin_repair manually if someone can confirm the safe sequence, or removing/recreating lvol0_pmspare if that’s appropriate here.

Full lvs -a output and journalctl available on request.

2

Follow the steps outlined this thread - All vms suddenly stopped working (cant start any vm, even sys-net) - #13 by passing_clouds