Why this matters beyond the homelab: Every environment eventually outgrows the memory of the person who built it. The practices in this act, decision records, docs-as-code, and machine-readable operational context, are the same ones that separate a team that can onboard an engineer in a week from one that cannot.
The Architecture covered how the Alliance Fleet was designed. The Act 2 posts covered what got deployed into it: SIEM, identity, observability, automation, roughly 25 services across three nodes and five VLANs.
Act 3 is different. Nothing in it is a new service. Every post in this act is about the same problem from a different angle: the fleet had grown past the point where I could hold it in my head, and nothing in it could explain itself.
That sounds like a documentation problem. It is actually an operations problem, and it announced itself the way these things usually do.
It Started With Losing Everything
On June 9 the n8n container came back from a rebuild with no workflows. Not corrupted, not partially restored. Thirty-eight workflows, gone from the UI, months of automation representing the connective tissue between nearly every service in the fleet.
The root cause was mundane: a Docker storage assumption I had never verified. The data was not where I believed it was, and I had believed it for long enough that the belief had stopped feeling like an assumption.
What made that incident the opening of an act rather than just a bad evening was the second-order realization. I could not reconstruct the workflows from memory, and I could not reconstruct them from documentation either, because the documentation described what the fleet did, not why any of it had been built that way. I had a wiki full of procedures and no record of decisions.
So I Started Writing Decisions Down
Architecture Decision Records read like enterprise ceremony until you have personally forgotten why you made a choice. The format is small: the context you were operating in, the decision you made, the alternatives you rejected, and the consequences you accepted.
The fleet now runs on ten ADRs, and the discipline changed how the lab evolves more than the documents themselves did. Writing down the alternatives you rejected forces you to have actually considered them. Several ADRs changed shape while I was writing them, which is the point.
This is also the least homelab-specific thing in the act. ADRs are standard practice in engineering organizations, and the reason they get skipped is always the same: the value arrives months after the cost.
Then I Replaced the Bot That Could Not Scale
BD-1 was my custom Discord bot, and by the time I retired it, it had logged 996 process restarts. It worked, in the sense that it responded, but every new capability meant more of my own glue code holding up more of my own assumptions.
Jocasta is its successor: the Hermes Agent framework, local-first inference with cloud elevation for the queries that need it, 76 loadable skills, and a hard budget cap so an agent loop cannot quietly spend money. The build surfaced a constraint I had not planned for, documented as ADR-010: the VRAM ceiling on the GPU node dictated model selection more than any benchmark did.
The lesson that carried forward was about ownership rather than architecture. Writing the framework yourself feels like control and is usually just maintenance you volunteered for. K-2SO and BD-1 both taught the same thing from opposite directions.
Then I Gave It Something to Read
An agent with no context about your infrastructure is a chatbot. The Fleet Codex pipeline closes that gap: BookStack exports nightly to Gitea, and the agent reads from Git rather than from a database it cannot introspect.
Docs-as-code is the part worth stealing. Once documentation lives in version control it gets diffs, history, and blame, which means you can answer when a decision changed and not just what it currently is. It also means the documentation is consumable by anything that can read a repository.
The pipeline nearly died to a Gitea 403 that had nothing to do with permissions and everything to do with a proxy configuration, which is documented in that post because it is the kind of failure that costs an afternoon and then never appears in any tutorial.
And Then the Hardware Reminded Me It Exists
None of the above matters if the physical layer is unhealthy. In one weekend in July, three separate fires: a duplicate IP silently breaking a VM through an ARP collision, a fleet-wide Proxmox kernel upgrade, and an NVIDIA DKMS failure that took the AI node’s GPU offline entirely.
The DKMS failure is the one that connects back to the rest of the act. A kernel upgrade broke the driver build, which took out the GPU, which took out local inference, which took out the agent that was supposed to help me diagnose things. Automation that depends on the fleet cannot be your only tool for repairing the fleet.
Key Takeaways
Documentation that cannot be queried is an archive, not a tool. The value of the Codex pipeline is not that the docs exist. It is that something other than me can read them.
Decision records are cheap to write and expensive to reconstruct. Ten ADRs took a few hours total. The n8n incident cost more than that in a single evening, largely because no record existed of why the container was configured the way it was.
Prefer a maintained framework to your own glue. 996 restarts was the number that finally made the case, but the argument was valid several hundred restarts earlier.
Keep one diagnostic path that does not depend on the automation. When the GPU went down it took the agent with it. Every layer added in this act is a layer that can fail.
Verify storage assumptions before you need them. Not the ones you wrote down. The ones you stopped noticing.
This post is part of the Alliance Fleet series on Holocron Logs. Act 3 in order: The Night n8n Forgot Everything, Writing ADRs for a Homelab, Jocasta, The Fleet Codex Pipeline, and One Weekend, Three Fires.