I’ve recently discovered an external drive has an extreme amount of bad sectors which required a different computer (running Ubuntu) to fix, as they were unfindable/unfixable within Qubes VMs. I will create a bug-report on this, but wanted to discuss it here first to figure out what to inspect/what logs to provide, etc.
I’m gonna describe my setup first, then what issues have occurred and how I fixed them in the end.
My setup
I use a USB-hdd for large data-storage. It contains one LUKS-partition with an ext4 filesystem. It is usually qvm-block attached to a disposable VM where it is decrypted. This exposes a new block-device dm-0. dm-0 is then attached to other VMs to run backups, download large files directly to the hdd. The decrypted partition usually stays attached to some VM when the computer goes to sleep and because of my hardware, I can only use S0ix sleep.
The issues
I’ve had many issues and can’t really tell which have to do with the broken file-system. Since the update to Qubes 4.3, a whole range of usb-issues appeared which I’ve collected here: General device issues after installing 4.3 · Issue #10636 · QubesOS/qubes-issues · GitHub
Especially the issue regarding transfer-speed I suspect is related to broken sectors rather than a qubes-specific USB-issue: USB transfer speed is abysmal · Issue #10675 · QubesOS/qubes-issues · GitHub
My duplicity-backup complained that one of the incremental backup-files was corrupted and unrecoverable. I used a torrent-program to download some files to the HDD and used it to recheck the files it had downloaded, about half of which reported unrecoverable files at seemingly random points.
However running fsck on the decrypted partition didn’t show any issues. I’ve tried forcing a full check from within a VM, but I aborted that check after ~3 weeks of extremely slow progress. (It made progress, but I calculated with the rate of progress at the time, it would take about 6 months to finish)
The solution
I’ve used an old laptop with Ubuntu to decrypt the drive and run fsck. It immediately found [more issues than fit on a single screen], but out of curiosity I ran the full check as well and it finished within ~2 days.
I haven’t used the hdd for long enough with Qubes again to make any definitive statements yet, but so far everything seems to be working better, apart from some missing bits where sectors where repaired
Causes?
I suspect some file-access getting interrupted when sleeping is causing the broken sectors. The sleep-issue didn’t exist about 2.5 years ago, so it’s possible the drive I’ve used before just didn’t accumulate enough errors to stand out.
I don’t think my HDD is physically broken. It’s less than 2 years old and a spiny-disk. Furthermore, I’d expect the LUKS-decryption to break instead of the ext4 filesystem.
When I report a bug on this, what are the relevant parts? The most unnerving part to me was that fsck didn’t report any issue on Qubes. But as the drive seems to work now, I can’t reproduce the exact output anymore.