Why this matters beyond the homelab: Stateful container storage failures are one of the most common production incidents in containerized environments. Verifying persistence assumptions before you need them is the difference between a restore and a rebuild.
The Incident
n8n is the automation glue of the Alliance Fleet. Nightly documentation exports, Discord alerting, job search automation, SIEM enrichment. When I rebuilt the container after a routine update, the UI came back clean. Not clean as in healthy. Clean as in empty.
Zero workflows. Zero credentials. Zero execution history.
38 workflows representing months of iterative automation work were no longer visible to the application. The first sixty seconds of an incident like this are about resisting the urge to “fix” anything before you understand what actually happened, because the wrong fix here (recreating the container again, reinitializing the database) is exactly how recoverable data becomes unrecoverable.
Timeline
- T+0: Container rebuild completes. n8n UI loads with no workflows.
- T+5m: Confirmed the container was healthy. This was not an app failure, it was a data location failure.
- T+15m: Inspected the container’s storage configuration. Found the problem: n8n’s data directory was writing to an anonymous Docker volume, not the bind mount path I believed it was using.
- T+30m: Located the orphaned volume from the previous container generation. The SQLite database with all 38 workflows was intact inside it.
- T+1h: Corrected the compose definition to use an explicit bind mount to host storage. Verified the path with a test write before restoring anything.
- T+2h: All 38 workflows reimported and validated. Credentials re-entered. Executions confirmed firing.
Root Cause
The compose file defined a volume, but the path syntax created a named Docker volume instead of the host bind mount I intended. Every workflow I had built was being written to Docker-managed storage that did not survive the way I assumed it would across a rebuild done with cleanup flags.
The deeper root cause is more uncomfortable: I had never tested the restore path. The data was “persisted” in the sense that it existed, but persistence you have not verified through an actual restore is a hypothesis, not a backup strategy.
Resolution
Three changes came out of this incident:
- Explicit bind mounts for every stateful container. Absolute host paths, no ambiguity, verified with
docker inspectafter deployment. If I cannotlsthe data from the host, it does not count as persisted. - Workflow exports as code. n8n workflows now export to JSON on a schedule and land in Git. The application database is no longer the only copy of the automation logic. This is the same principle as infrastructure-as-code: the running state should be reproducible from version control.
- Restore drills. The Alliance Fleet backup architecture (Proxmox snapshots, NFS, Azure offsite via azcopy) already existed when this happened. It did not help, because n8n’s data was not in any of those paths. Backup coverage is now audited against a service inventory, not assumed.
Key Takeaways
- An untested backup is a hypothesis. The restore is the backup.
- Docker volume syntax is a production risk surface. Anonymous volumes and bind mounts look nearly identical in a compose file and behave completely differently during teardown.
- Coverage audits beat coverage assumptions. My offsite pipeline was solid and completely irrelevant, because the failing service was never enrolled in it.
- The incident cost two hours. The process changes it forced have already paid that back.