Last week write access to my NVMe ssd running Qubes 4.3 was lost with data corruption (on the dom0 root volume) preventing boot. Installing Qubes on another NVMe ssd and proceeding to debug, I determined the data corruption was too pervasive. I successfully made disk images of my various private volumes (no recent backups, oops). After I finished that, my fresh Qubes install on the second NVMe ssd crashed in the same way, still booting but with some data corruption. On subsequent boots it continued to crash in the same way. Switching to plain Debian Trixie on one of the drives resolved the crashing issue so I suspect I’ve run into a regression in Qubes or Xen.
Right now I don’t have access to logs since the systems couldn’t write any information after the drives crashed and there doesn’t appear to be anything leading up to the problem in them. I’ll figure out a way to produce the problem again and preserve logs, could someone point out which ones in particular I should gather?
Hardware:
Originally an Asrock X670E Taichi, tried again with the same result with an Asrock X870E Taichi
Just realized I didn’t define “crash”. I mean the system (dom0, qubes) suddenly lose the ability to write to the disk, the mount ends up R/O with a ton of kernel errors from the filesystem (ext4) not being able to write caches to disk etc. I’ll get log outputs as soon as I can.
Yes, and in my case the only path forward for me was to ditch the LVM filesystem altogether and to reinstall on top of BTRFS instead. I tried installing with the default LVM filesystem option four times, on top of brand-new Samsung SSD’s, and there was no joy each time. Each time I was unable to restore from backup because of this remounting RO problem with some temporary space used during that restore. After having so many issues with installing on the new drives I ran the extended S.M.A.R.T tests on the new drives, and it reported nothing was wrong with them. With BTRFS it runs and restored just fine, and I have seen absolutely no hardware issues since.
This is very strange because this same machine was running just fine for many years on top of LVM/SSD’s but then started having problems with the drives, so I replace them and I just could not get the new SSD’s to cooperate. The older drives had legitimately too many entries in the bad block table so they needed to be replaced, but the drives themselves had not actually failed outright. It was time to replace them anyway, but I can’t help but wondering if some lower level bug didn’t hurry them along to their demise. I don’t know enough about the low level LVM architecture to even guess what might have gone wrong with LVM on the new drives.
Oops, I was coming to the same conclusion thinking about this over night. It might be possible I ran into an actual bug (or hardware fault) on my first SSD, then ran out of space without realizing it while recovering data on the second, smaller SSD. Or perhaps I ran into the same issue on both. Going to have to do some more digging on my end.