Why this matters beyond the homelab: Storage incidents are where “we have backups” gets tested and frequently fails. This one involved reading failure symptoms correctly instead of trusting a health check, choosing a recovery order that preserved evidence, and then going back the next day to find the actual root cause rather than declaring victory at “it boots now.” That sequence is the job.
What Happened
I ran a docker compose pull on Yavin-IV, the VM hosting the Fleet Codex stack (BookStack, Gitea, NetBox, and their databases). Ordinary maintenance. Ten containers, a pull, a recreate.
The recreate failed. The console filled with EXT4 I/O errors and journal aborts. Within seconds the underlying LVM-thin pool on Node-B had tripped into metadata_read_only, which is thin-provisioning’s way of saying it no longer trusts its own bookkeeping and would rather stop than make things worse.
The VM was down. The Codex, meaning my documentation, my wiki, my Git remote, and my source of truth for the whole fleet, was inside it.
The First Useful Surprise
Standard triage says check the drive. So I did:
smartctl -a /dev/nvme0n1
The drive was fine. Zero media errors. Spare capacity at 100%. No reallocations, no pending sectors, no warnings. By every metric SMART reports, this NVMe was healthy.
That result is easy to misread in both directions. It does not mean nothing is wrong, and it does not mean the drive is innocent. What it actually means is that the failure is not in the storage media. Something in the write path was failing while the media itself was intact.
This distinction turned out to be the entire incident.
Reads Worked
Before doing anything destructive, I tested whether the volume could still be read at all:
dd if=/dev/mapper/<thin-volume> of=/dev/null bs=1M count=1024 status=progress
It read at full speed. No errors.
Writes, meanwhile, were failing. Every attempt to repair the filesystem in place errored out.
That asymmetry, a disk that reads perfectly and cannot write, is the best possible version of this failure. It means the data is still there and intact. It means you have exactly one move available, and it is the right one.
Recovery, In Order
The order mattered more than any individual command.
Step one: get a clean copy off the failing disk before touching anything.
ddrescue -f -n /dev/mapper/<source-thin-volume> /mnt/rescue/yavin-iv.img /mnt/rescue/yavin-iv.log
This completed in under seven minutes at 100%. No bad sectors, no retries, no scraping pass needed. Because reads were unaffected, ddrescue behaved like a straight dd with a logfile.
The logfile matters even when the copy is clean. If the run had been interrupted, it would have resumed rather than restarting, and it documents exactly which regions were recovered.
Step two: repair the image, not the disk.
I had already attempted e2fsck twice against the original volume. Both attempts failed, because repairing a filesystem requires writing to it, and writing was the broken half.
Against the rescued image on healthy storage:
e2fsck -y -f /mnt/rescue/yavin-iv.img
It succeeded cleanly on the first attempt.
Same filesystem. Same corruption. Same tool. The only variable was the storage underneath it. If you take one thing from this post: a repair tool cannot fix a filesystem on a device that cannot accept writes, and its failure tells you nothing about the filesystem itself.
Step three: verify before trusting.
Before repointing anything, I mounted the repaired image read-only and confirmed all ten Docker named volumes were present with their data intact. Verifying the payload before rebuilding around it is cheap. Discovering an empty volume after cutover is not.
Step four: migrate to healthy storage.
The repaired image was copied onto a fresh thin volume on the fast-lvm pool, and VM 500’s scsi0 was repointed there. The guest booted clean.
Nine of ten containers came back healthy immediately. The tenth, NetBox’s housekeeping worker, needed a coordinated docker compose up -d rather than a bare container restart, because it depends on startup ordering that a single-container restart does not respect.
Zero data loss.
The Root Cause, Found the Next Day
The temptation at this point is enormous: it boots, the containers are green, close the ticket.
I went back the following afternoon, and the real finding was waiting in the boot history:
journalctl --list-boots | wc -l
smartctl -a /dev/nvme0n1 | grep -i "unsafe shutdown"
Nine unclean crashes since January. Ninety-five unsafe shutdowns on the drive.
This reframed everything. The incident was not a disk spontaneously failing during a docker compose pull. It was the accumulated cost of months of hard crashes finally landing on a write that mattered. The pull was not the cause, it was just the unlucky moment when the bill came due.
The crash trigger itself remains unconfirmed. An r8169 NIC fault interacting with IOMMU is the leading suspect based on what little log evidence survived. The current boot has shown zero recurrence, which is not the same as being fixed, and I am treating it as open.
Why Root Cause Took So Long to Even Attempt
Because the logs were gone.
journald was not configured for persistent storage, so every reboot wiped the evidence of why the previous boot ended. Nine crashes had happened, and I had forensic data for none of them.
Fixed immediately:
mkdir -p /var/log/journal
systemd-tmpfiles --create --prefix /var/log/journal
systemctl restart systemd-journald
This is the second time an observability gap has cost me a root cause in this lab. It is a cheap fix and I should not have needed a second lesson.
The Finding That Actually Stung
VM 500 had never been backed up. Not once in its history.
The VM holding the wiki, the Git remote, and the network source of truth for the entire fleet was outside every backup job I run. I have a nightly vzdump schedule. I have an Azure offsite pipeline. Neither of them included this machine, because it was created after the jobs were defined and nobody, meaning me, went back to add it.
The recovery succeeded on the strength of a favorable failure mode. Reads happened to still work. Had this been a media failure instead of a write-path failure, ddrescue would have returned a partial image full of holes and the Codex would be gone.
That is not a recovery strategy. That is luck.
VM 500 was added to the Node-B backup job the same afternoon, and I audited every guest against the backup schedule rather than assuming coverage.
Takeaways
SMART health is not system health. A perfectly healthy drive can sit behind a broken write path. Read the failure symptom, not just the self-report.
Test reads and writes separately. The asymmetry determines what recovery options exist. Reads-work-writes-fail is recoverable. Neither-works is a restore.
Repair the image, never the failing device. e2fsck against a copy on good storage succeeds where the same command against the original cannot, and the difference has nothing to do with the filesystem.
Verify the payload before rebuilding around it. Confirm your volumes exist before you repoint disks.
“It boots” is not root cause. The mechanism was accumulated crash damage, not a bad pull. Without going back the next day, I would have learned nothing and hit it again.
Persistent logs are prerequisite infrastructure. Nine crashes with no surviving evidence is a self-inflicted blind spot.
Audit backup coverage against reality, not against intent. The gap was not in the schedule design, it was in the assumption that new guests get added to it. They do not. Check.
Related: The Alliance Fleet Backup Architecture | One Weekend, Three Fires | Diagnosing a Silent Crash With No Logs