Make a copy or backup of the entire disk (dd it to a file on external storage). In most cases, the AI isn’t always right, and you’ll need to try several times.
Response from Opus 5
A few things in your post narrow this down a lot, and I think the direction suggested above is dangerous on your system specifically. Please don’t run the vgcfgrestore config-edit yet.
Your two errors are not the same problem. The create_snap 4129 3393 one is the queued message (and for the record, the kernel parses that as create_snap <new_dev_id> <origin_dev_id> — 3393 is the origin’s device id, not a transaction id). But the failure that’s actually stopping your VMs is the other one:
device-mapper: reload ioctl on (253:XX) failed: No data available
That comes from thin_ctr → dm_pool_open_thin_device() → __open_device(), which does a btree lookup in the pool’s details tree and returns -ENODATA when the dev_id simply isn’t there. So each of those means “this LV’s device id is not present in the metadata currently attached to the pool.”
Which is why the advice to delete the broken LV definitions from vgcfgbackup would be very bad here. You’re seeing this across dozens of unrelated volumes, different VMs, different ages, both -root and -private. One uncommitted create_snap accounts for exactly one new device id — it cannot make dozens of long-existing volumes vanish. So those LVs aren’t phantom entries to be pruned; they look like real volumes whose metadata records are missing. Editing them out of the VG config would make that permanent.
The meta0–meta6 sequence is the thing I’d focus on. Per lvmthin(7), each lvconvert --repair run displaces the previous metadata to ThinPool_metaN. Seven of them means the repair ran seven times, and thin_repair only recovers the devices it can still reach — so runs 2–7 each rebuilt from the already-degraded output of the run before. If that’s what happened, vm-pool_meta0 is the only copy with genuine pre-crash metadata, and the fix is to restore it rather than to prune the config. Please don’t run --repair again; each run costs another generation.
Read-only commands to confirm this before anyone touches anything. These write nothing to the pool:
# 1. Full LV inventory + the device ids LVM expects
sudo lvs -a -o lv_name,lv_attr,lv_size,pool_lv,thin_id,origin qubes_dom0
sudo lvs -a --noheadings -o thin_id qubes_dom0 | grep -c '[0-9]'
sudo vgs qubes_dom0 # also shows whether you have free space for a new metadata LV
# 2. Device count in every metadata copy you still have
for m in $(sudo lvs --noheadings -o lv_name qubes_dom0 | tr -d ' ' \
| grep -E '^vm-pool_(meta[0-9]+|repaired_meta)$'); do
sudo lvchange -ay qubes_dom0/$m 2>/dev/null || { echo "$m: cannot activate"; continue; }
sudo thin_dump /dev/qubes_dom0/$m -o /root/dump_$m.xml 2>/dev/null
echo "$m devices=$(grep -c 'dev_id=' /root/dump_$m.xml) \
$(grep -m1 -o 'transaction="[0-9]*"' /root/dump_$m.xml)"
sudo lvchange -an qubes_dom0/$m
done
# 3. The pool's queued message and transaction_id as LVM sees them
sudo vgcfgbackup -f /root/vg.txt qubes_dom0
sudo grep -n 'transaction_id\|message[0-9]\|create =\|delete =' /root/vg.txt
sudo dmesg | grep -iE 'thin|device-mapper' | tail -50
Note thin_dump deliberately without -r here — we want to see what’s actually in each copy, not a repaired view of it.
What the output decides:
meta0 has roughly the full device count and the live tmeta is far short → the repairs ate your metadata. Fix is thin_restore the meta0 dump into a fresh LV and swap it in with lvconvert --thinpool qubes_dom0/vm-pool --poolmetadata qubes_dom0/vm-pool_newmeta, then clear the stale message1 { create = "..." } block and align transaction_id. Your data blocks in tdata should be untouched — thin_repair only ever writes metadata, and nothing has been able to start and allocate.- Every copy including
meta0 is equally short → the loss predates all the repairs and restoring meta0 gains nothing; that would point at the tmeta’s underlying storage, and the next step is a different investigation entirely.
- The counts actually match what
lvs expects → then this isn’t missing devices at all and the whole theory above is wrong; we’d be looking at the pool sitting in fail/read-only mode instead.
Update 2026-09-01T11:53:00Z
Response from Opus 5
Good catch — my “these write nothing to the pool” was too strong. Auditing them honestly:
Genuinely read-only: lvs, vgs, grep, dmesg. These take read locks and touch nothing.
Not read-only — changes kernel state: lvchange -ay / -an. Activation creates and tears down device-mapper mappings. It doesn’t write to the pool’s data or metadata, and it doesn’t modify VG metadata on disk, but calling it “read-only” is wrong. It’s non-destructive, not passive.
Not read-only — writes files: thin_dump -o /root/dump_$m.xml and vgcfgbackup -f /root/vg.txt. Both only read the LVM devices, but they write output to dom0’s filesystem. Two concrete risks I glossed over:
- A full
thin_dump of a Qubes pool’s metadata can be hundreds of megabytes to multiple gigabytes of XML, because it emits every block mapping range. Doing that seven times could fill dom0’s root.
- If dom0’s root filesystem happens to live in a thin pool, writing large files into it consumes pool space — exactly the wrong thing to do while a pool is in trouble.
Both are avoidable. For the survey we only need device ids and the transaction id, not the mappings, and thin_dump --skip-mappings cuts the output to a few KB. Better still, write to external media rather than /root.
Two things that are safe by construction and worth keeping: the loop uses lvs without -a, so it only ever sees visible LVs and cannot accidentally grab the metadata LV currently attached to the pool. And it only dumps the detached _metaN copies, never the live tmeta — which matters, because thin_dump(8) says it “cannot be run on live metadata unless the --metadata-snap option is used,” and running it on live metadata reports corruption that isn’t real.
Revised step 2 for the post:
DEST=/mnt/usb # external media, not dom0 root
for m in $(sudo lvs --noheadings -o lv_name qubes_dom0 | tr -d ' ' \
| grep -E '^vm-pool_(meta[0-9]+|repaired_meta)$'); do
sudo lvchange -ay qubes_dom0/$m 2>/dev/null || { echo "$m: cannot activate"; continue; }
sudo thin_dump --skip-mappings /dev/qubes_dom0/$m -o $DEST/dump_$m.xml 2>/dev/null
echo "$m devices=$(grep -c 'dev_id=' $DEST/dump_$m.xml) \
$(grep -m1 -o 'transaction=\"[0-9]*\"' $DEST/dump_$m.xml)"
sudo lvchange -an qubes_dom0/$m
done
And the honest framing for the post is “nothing here writes to the pool’s data or metadata” rather than “read-only” — the distinction being that activation is reversible state, and file writes go somewhere you’ve chosen deliberately.
