Why this matters beyond the homelab: Configuration drift between what is declared and what is running is one of the most common sources of production incidents, and it is almost always self-inflicted during maintenance. The containers looked healthy. The orchestrator was satisfied. The only thing wrong was invisible unless you asked a question nobody thinks to ask.
What Broke
Two symptoms, seemingly unrelated.
The Samba share stopped serving. The arcade share, which hosts BIOS files, ROMs, saves, and media for the retro gaming VM, was unreachable.
Separately, Authentik’s worker process stopped processing tasks. The login page worked fine. Users could authenticate. But background tasks queued and never ran, which is the kind of failure that hides for a while because the visible half of the service is still up.
Two different services, two different symptoms, one cause.
The Cause
During earlier maintenance, Redis and Samba had been recreated using bare docker run commands rather than being brought up through their actual compose stack at /opt/stacks/postgres/docker-compose.yml.
Both containers started. Both reported healthy. Both were on the default bridge network instead of postgres_default, the network the rest of the stack lives on.
That is the entire incident. A container on the wrong network is not a broken container. It runs perfectly, passes its health check, shows green in Portainer, and cannot be reached by name from any of its peers.
Authentik’s worker depends on Redis for its task queue. Redis was up, healthy, and unreachable, so the worker had nothing to talk to. Samba was similarly isolated from the stack it belonged to.
The docker run invocations had probably been correct at the time in every way except network membership, which is easy to omit because compose handles it invisibly and you never think about it.
Confirming It Rather Than Guessing
The hypothesis was network isolation. Confirming it takes one command per container:
docker inspect redis --format '{{json .NetworkSettings.Networks}}' | jq
docker inspect postgres --format '{{json .NetworkSettings.Networks}}' | jq
Redis showed bridge. Postgres, which had never been touched, showed postgres_default. Same for Samba.
Diffing the network membership of a known-good container against a suspect one turns “I think it is the network” into a confirmed root cause in about ten seconds. This is worth doing before making changes, because otherwise you fix it, it works, and you never actually know what was wrong.
The Recovery Decision That Mattered
Here is the part I want to highlight, because it is the difference between a clean recovery and a subtly wrong one.
Samba needed to be recreated on the right network. To do that correctly I needed its original configuration: the share name, the user, the mount path, the exposed directories.
I could remember roughly what it was. I could have written a compose service from that memory, brought it up, confirmed the share mounted, and moved on. It would have appeared to work.
Instead I went and found the original. Portainer stores the compose file for every stack it manages on disk:
cat /data/compose/10/docker-compose.yml
There it was, exactly as originally deployed: share named arcade, user arcade, serving /srv/arcade, with four data directories underneath (bios, media, roms, saves).
My memory was close. It was not exact, and “close” on a Samba share configuration means permissions or paths subtly different from what every client expects, discovered weeks later when something cannot find its save files.
When the real configuration still exists somewhere, find it instead of reconstructing it. Portainer’s stack storage, a Git remote, a backup, another node running the same service. Reconstruction from memory during an incident is how one outage quietly becomes a second one.
The Fix
Not docker run with the right --network flag. That would have restored service while leaving the same drift in place for next time.
The fix was bringing the whole stack up through compose, from the real compose directory:
cd /opt/stacks/postgres
docker compose down
docker compose up -d
Then verifying the data layer before trusting anything above it:
docker exec -it postgres psql -U authentik -c '\dt'
Schema intact. Only then did I check whether Authentik’s worker had recovered, because confirming a downstream service works while its database is quietly damaged just moves the incident later.
Order of verification: network membership, then database integrity, then dependent services. Bottom up, every time.
Why It Happened At All
Not ignorance of compose. Convenience under pressure.
Mid-maintenance, a container needs to come back, and docker run is one line you can type from memory while docker compose up requires being in the right directory with the right file. The shortcut works often enough to become a habit, and it silently discards everything the compose file declares: networks, volumes, environment, restart policy, dependency ordering, labels.
The container that results is similar to the declared one. Similar is the problem. A container that failed to start would have been caught in seconds. A container that starts successfully in a slightly wrong shape survives until something downstream needs the part that is missing.
The rule I now follow without exception: if a service is defined in a compose file, it only ever comes up through that compose file. No exceptions for “just this once,” because “just this once” is exactly what caused this.
Takeaways
Healthy is not correct. A container can pass every health check while being unreachable by the services that need it.
Diff network membership against a known-good peer. docker inspect confirms in seconds what would otherwise be a guess.
Never reconstruct config from memory when the original is recoverable. Portainer keeps stack files on disk. Look there first.
Bring stacks up through compose, always. docker run drops networks, volumes, restart policy, and ordering without telling you.
Verify bottom-up. Network, then database schema, then dependent services. Confirming an app works on top of a damaged database just delays the incident.
Half-up services hide. Authentik’s login page worked the entire time. Check the components that do not have a UI.
Related: Portainer: Container Orchestration Across a 3-Node Proxmox Cluster | PostgreSQL and Redis: The Data Backbone | The Night n8n Forgot Everything