Why this matters beyond the homelab: The interesting failure is not that a service went down. It is that it stayed down for two weeks in an environment with a SIEM, a metrics stack, and a dedicated uptime monitor. Coverage gaps are invisible by definition, and you find them by auditing inventory against monitoring rather than by waiting for an alert that structurally cannot fire.
The Discovery
While working on something unrelated, I checked the network documentation service and found it unreachable. Six containers in the stack, all stopped.
The last log entries were two weeks old, timestamped at a host reboot.
Two weeks. In a lab with fifteen SIEM agents, a metrics pipeline scraping everything, and an uptime monitor covering twenty-five-plus services. Nothing had said a word.
Why It Never Came Back
The stack has a dependency chain: an application, a background worker, a Postgres database, a cache, and supporting containers.
The application containers had restart: unless-stopped. The Postgres and cache services did not.
After the reboot, Docker restarted the containers that had a restart policy and left the ones that did not. So the application came up. Its database did not.
That produced a state that is worse than a clean outage: the application containers were running, retrying their database connection, logging timeouts, and waiting for something that was never going to start. From the outside, containers existed and had a process in them. The background worker in particular sat in a retry loop timing out against a database that was not there.
A stopped stack is obvious. A half-started stack that is busily failing looks like activity.
Why Nothing Alerted
Three separate gaps, each independently sufficient to produce two weeks of silence.
It was not in the uptime monitor. Twenty-five services were being checked. This was not one of them. It had been deployed after the monitoring config was built and never added, exactly the same way a VM got created after a backup job was defined and never joined it. Same class of gap, different system.
The SIEM agent reported the host, not the service. The host was up and healthy, because the host genuinely was up and healthy. Host-level monitoring cannot tell you that a container on that host is missing.
Container health checks did not help. The containers that were running had processes in them. Without an application-level check that actually queries the service and expects a valid response, “a process exists” is all you learn, and a process stuck in a database retry loop satisfies that trivially.
Every layer was reporting accurately within its scope. The scope was the problem.
The Fix
Restart policies on every service in the stack, not just the ones I thought of as the application:
services:
netbox:
restart: unless-stopped
netbox-worker:
restart: unless-stopped
postgres:
restart: unless-stopped # was missing
valkey:
restart: unless-stopped # was missing
The infrastructure services underneath are the ones that must come back, because everything above them depends on them. They are also the easiest to overlook, because you interact with the application and rarely think about its database as a thing that needs a policy.
Then verify it survives a reboot, rather than assuming:
docker compose up -d
reboot
# after boot
docker compose ps
An untested restart policy is a claim. The claim was already wrong once.
Add the service to the uptime monitor, with a check that queries an application endpoint rather than pinging the host.
Then audit everything else the same way, because if one service was missing from monitoring, others were. I enumerated running services and diffed that list against the uptime monitor’s configured checks. It found more than one gap.
The Pattern I Keep Hitting
This is the third time in a short period that the same underlying mistake produced a different incident:
- A VM created after the backup jobs were defined, never backed up.
- A service deployed after the monitoring config was built, never monitored.
- A container fix applied at runtime instead of in the compose file, silently reverted.
All three are the same failure: infrastructure defined at one point in time, then extended without revisiting the systems that are supposed to cover it. Backup jobs, monitoring configs, and declarative sources do not discover new things on their own. Somebody has to add them, and if adding them is not part of provisioning, it does not happen.
The fix that generalizes is a provisioning checklist. Anything new gets added to backup, added to monitoring, and defined in a tracked file, at creation time rather than whenever the gap eventually causes an incident.
The fix that catches what the checklist misses is periodic reconciliation. Enumerate what exists, diff it against what each covering system knows about, investigate every difference. It is not exciting work and it finds real problems every time.
Takeaways
Restart policies belong on infrastructure services, not just applications. The database is the one that has to come back.
Test restart policies with an actual reboot. Untested means unverified.
Half-started stacks hide. A container retrying a connection looks alive.
Health checks need to query the application. Process-exists is not service-works.
Host monitoring does not cover service monitoring. Both layers are needed and they answer different questions.
Monitoring configs go stale silently. Nothing alerts you about a missing check, by definition.
Reconcile inventory against coverage periodically. Enumerate, diff, investigate.
Related: Uptime Kuma: Monitoring 25+ Services | The Fix That Unfixed Itself | Auditing a Backup Strategy I Thought I Already Had