← Holocron Logs

I Said Act Three Had No New Services. Then I Deployed Four.

Act 3 opened with a line I meant at the time: nothing in it is a new service. A month later I deployed Vikunja, Infisical, NetAlertX, and Outline. Every one of them exists because the stabilization work found a gap that could not be documented away, and this is the argument for each, plus a full accounting of what is actually running now.

Why this matters beyond the homelab: Every environment faces the same tension between adding capability and being able to operate what you already have. The useful discipline is not refusing to grow, it is having a defensible reason for each addition and knowing what it costs to run. This post is an audit of my own rule, including where I broke it.


The Line I Wrote

From Act Three: Making the Fleet Legible, published July 27:

Act 3 is different. Nothing in it is a new service.

I meant it. Act 2 had filled three nodes with roughly twenty-five services, and I had reached the point where I could not hold the fleet in my head. Adding more was clearly not the answer. The whole act was about documentation, decision records, an agent that could read the docs, and admitting that the physical layer needed attention.

Within a month I deployed four new services.

I want to be precise about why that is not simply me failing to follow my own rule, and equally precise about the part where it partially is.


What Changed the Calculation

Act 3 ended and I moved into a structured remediation effort: a twelve-sprint plan to patch, verify, and harden everything already running before doing anything else. Sprint 01 alone touched seventeen guests.

That work found problems. Not the interesting kind, the boring kind that compound:

Credentials were everywhere. During the Wazuh dual-auth incident I discovered a dashboard authenticating with an install-time password from months earlier, because a credential had been rotated in one place and never propagated. That was one instance of a general condition. Long-lived tokens sat in env files. A storage account SAS token with a distant expiry lived on disk in a cron job. API tokens for the observability bot were scattered across four services with no single record of what existed or when it expired.

I could not enumerate my own network. NetBox documents what I told it about. It has no idea what is actually on the wire. When a duplicate IP caused an ARP collision in July, nothing detected it, because nothing was watching for devices appearing that I had not declared.

Work was tracked in my head and in Discord scrollback. A twelve-sprint plan is not something you hold in working memory, and I proved that by losing track of which sprint items were done.

Documentation had split into two audiences. The wiki is the machine-readable source that feeds the agent through the Codex pipeline, and it needs to stay structured. But I also needed somewhere to think in prose, and I had been polluting the structured wiki with half-formed notes.

Each of those is a gap that documentation cannot close, because each one is a missing capability rather than a missing description. That is the distinction I settled on, and it is the rule I would apply going forward: documentation fixes not knowing. It does not fix not being able.


The Four, and the Case for Each

Infisical: secrets management

The strongest justification of the four, and the one I would defend hardest.

The problem was not that credentials were insecure in any dramatic sense. It was that they had no lifecycle. No record of what existed, no rotation process, no expiry tracking, no audit trail of what read them. The failure mode is exactly what bit me with Wazuh: change a credential in one place, forget the three consumers, discover months later that something has been quietly broken.

Infisical gives centralized storage, versioning, machine identities for services rather than shared credentials, and an audit log. Services pull secrets at runtime instead of holding copies.

The rollout is deliberately phased, low-stakes services first, because a secrets manager becomes a single point of failure the moment anything important depends on it. I am not migrating the identity provider’s credentials until I have watched it behave for a while.

The certification angle is real and I will not pretend otherwise: RBAC, versioning, and machine identities map closely onto Azure Key Vault concepts, and I am working through SC-300. But the credential sprawl was a genuine problem before the cert track made it convenient.

NetAlertX: network discovery

This one exists because of a specific incident. An ARP collision from a duplicate IP silently broke a VM in July, and it took manual investigation to find. The fleet had a SIEM, a metrics stack, and an uptime monitor, and none of them were watching for unexpected devices on the network, because that is a different question from are my known services healthy.

NetAlertX scans and alerts on devices appearing that were not there before. It closes the loop between what NetBox says should exist and what is actually connected, which is the same gap that let a service sit down for two weeks unnoticed: a monitoring system can only report on what it knows to look for.

Vikunja: task tracking

The weakest technical justification, and the one I would cut first if I had to.

The honest case: a twelve-sprint remediation plan across seventeen guests is real project management, and I was doing it out of Discord scrollback and memory. That failed. Sprint items got dropped, and the service that had been down for two weeks was arguably a symptom of having no system for tracking follow-ups.

The honest counter-argument: this is a workflow tool, not infrastructure. It has a REST API and it integrates with the bots, which is a nice property, but I could have used any hosted task tracker at zero operational cost. I chose to self-host it because self-hosting is the point of the lab, and that is a preference rather than a requirement. Worth naming as such.

Outline: prose documentation

The subtlest of the four and the one that took the most thinking.

BookStack is not just a wiki here. It is the source for the Codex pipeline, exporting nightly to Git so the agent can read it. That gives it a constraint most wikis do not have: its structure is an interface, and polluting it with unstructured thinking degrades what the agent retrieves.

I needed a separate place for prose. Drafts, incident narratives before they become postmortems, reasoning that has not resolved into a decision yet.

The counter-argument is legitimate: two documentation systems is exactly the kind of sprawl that produces the “which one is this in” problem. My answer is a hard rule about what goes where, structured operational reference in BookStack, unstructured thinking in Outline, and if I find myself unsure, it goes in BookStack. Whether that rule survives contact with six months of use is genuinely an open question, and I will write the follow-up either way.


How They Tie Into Everything Else

The four are not standalone. Each one plugs into machinery that already existed, which is most of the argument for adding them at this point rather than during Act 2.

Everything sits behind Authentik. SSO via OIDC means four new services and zero new login credentials. This is the compounding return on having built identity centrally: the marginal authentication cost of a new service is now approximately zero.

Everything is behind the reverse proxy, on the services VLAN, reachable remotely only over Tailscale. No new ingress, no new firewall exposure, no port forwards.

Everything got a Wazuh agent at deployment rather than as an afterthought, so they are in the SIEM from the first boot. Doing this at deploy time is a direct lesson from finding agents held back and silently stale on nodes where enrollment had been done later and sloppily.

Everything went into Uptime Kuma and onto the Homepage dashboard immediately, which is the specific correction for the two-week outage. Adding monitoring at provisioning time rather than “soon” is now a checklist item, not an intention.

Everything is in the backup job, which is the correction for discovering a VM that had never been backed up.

Infisical will eventually sit underneath all of it, which is the direction of travel: services stop holding credentials and start requesting them.

The pattern worth extracting: the fourth service is cheaper to add than the first, if the platform underneath is real. Identity, proxy, monitoring, backup, and logging were already solved problems, so each new deployment inherited them. That is why Act 3’s stabilization work made this expansion defensible rather than reckless. Adding four services to an environment I could not explain would have been the mistake. Adding four to one with decision records, a documentation pipeline, and a verified backup posture is a different act with the same name.


What Is Actually Running

A full accounting, since the point of this post is honesty about scope.

Network and access AdGuard Home (DNS filtering and internal resolution), Nginx Proxy Manager (reverse proxy and TLS), Tailscale (remote access), NetAlertX (device discovery) (new)

Identity and secrets Authentik (SSO via OIDC and SAML across the fleet), a Windows Server domain controller with hybrid sync to Entra ID, Vaultwarden (personal vault), Infisical (service secrets) (new)

Security and monitoring Wazuh manager, indexer, and dashboard with a fifteen-agent fleet and a 41-rule custom ruleset, Uptime Kuma (service availability)

Observability Telegraf, InfluxDB, and Grafana

Automation n8n (the command bridge workflows), Jocasta (the Hermes-based agent), K-2SO (the observability bot)

Documentation and knowledge BookStack (structured, feeds the Codex pipeline), Gitea (version control and the Codex remote), NetBox (network source of truth), Outline (prose) (new)

Platform and data Proxmox across three nodes, Portainer (container management), PostgreSQL and Redis (shared data backbone), Samba (file shares)

AI and compute Ollama, OpenWebUI, AnythingLLM, ComfyUI, on the GPU node

Productivity and other Homepage (unified dashboard), Home Assistant (isolated IoT VLAN), RustDesk (self-hosted remote desktop), Vikunja (task tracking) (new)

Roughly thirty services across three nodes and five VLANs.


The Honest Assessment

Three of the four close a capability gap that stabilization work identified. One of them, Vikunja, is a workflow preference I chose to self-host because self-hosting is the hobby. I would rather say that plainly than construct a technical justification for it after the fact.

The rule I am actually operating under now, stated so I can be held to it:

A new service needs a gap that documentation cannot close. If the problem is “I forgot how this works,” write it down. If the problem is “I cannot see this,” that might be a service.

It has to inherit the platform on day one. SSO, proxy, monitoring, backup, SIEM agent, at deployment time. If it is not worth ten minutes of integration, it is not worth running.

Adding it means owning it. Every service is another thing to patch, another set of credentials, another dependency in an incident. Sprint 01 took seventeen guests to complete. It is now larger.

The count is not the metric. Thirty services I can patch, monitor, back up, and explain is a healthier environment than fifteen I cannot. Act 3 was not about being smaller. It was about being legible, and legibility is what made this expansion affordable.

I will revisit this in six months and note which of the four are still earning their place. My guess is that Infisical and NetAlertX will be obviously load-bearing, and that one of the other two will have quietly become something I do not open.


This post is part of the Alliance Fleet series on Holocron Logs. Prior context: Act Three: Making the Fleet Legible, The Fleet Manifest, Writing ADRs for a Homelab.

← Back to Holocron Logs