Why this matters beyond the homelab: Configuration drift between declared state and running state is silent, patient, and reverses your fixes without notification. The incident here is small. The class of mistake is not, and it applies to every declarative system: Docker Compose, Kubernetes manifests, Terraform, Ansible.
February
The AI node runs two GPU workloads: an inference server for local language models, and an image generation service. Both want the same card.
Running both under sustained load simultaneously locked the entire node up. Not a container crash, not a service failure. The host went unresponsive and required a hard power cycle.
The fix at the time was straightforward: the image generation service should not come back automatically. It is used interactively, so there is no reason for it to restart on its own and re-enter contention for a GPU that the inference server needs continuously.
I changed the restart policy, verified the behavior, wrote it up in the documentation as a known hazard with a stated remediation, and moved on.
August
Six months later, during a maintenance pass, I was checking GPU-adjacent services before upgrading the container toolkit.
The image generation service was running with:
restart: unless-stopped
The exact configuration that caused the February lockup. Present, active, and having been that way for an unknown length of time.
Nothing had broken since. The two services had simply not hit sustained simultaneous load, so the hazard sat there without consequence. If I had run the toolkit upgrade without checking, a service restarting mid-upgrade into GPU contention would have been an excellent way to reproduce February from scratch.
Why It Reverted
I had fixed the running container. I had not fixed the compose file.
# what I did in February, roughly
docker update --restart=no comfyui
That works. It changes the live container’s restart policy immediately and it survives reboots. Verification passes. The problem is solved, right up until the container is recreated.
And containers get recreated constantly during normal operations. A docker compose up -d after an image pull. A stack redeploy through the orchestrator. Any change to any service in the same compose file. Every one of those regenerates the container from the compose file, which still said unless-stopped, because I never touched it.
The compose file is the source of truth. Anything you change on a running container is a temporary override with an expiry date determined by the next recreate, and you will not be told when that happens.
February’s fix was never a fix. It was a suppression with an unknown timer.
The Actual Fix
Change the declaration:
services:
comfyui:
image: ...
restart: "no"
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities: [gpu]
Then redeploy so running state and declared state converge:
cd /opt/stacks/ai
docker compose up -d comfyui
docker inspect comfyui --format '{{.HostConfig.RestartPolicy.Name}}'
That last line matters. Verify the running container reflects the declaration, rather than assuming the redeploy did what you intended. The whole failure mode here was a gap between declared and running state, so closing it out with an actual check rather than an assumption is the point.
Only after that did I proceed with the container toolkit upgrade, verifying GPU passthrough survived two full Docker daemon restarts before calling it done.
The Generalization
Every declarative system has this failure mode, and the shape is identical each time:
Docker Compose. docker update, docker run flags, manual docker network connect. All reverted by the next compose up.
Kubernetes. kubectl edit on a live resource, reverted the next time the manifest is applied.
Terraform. Console changes, reverted or fought on the next apply.
Ansible. Manual edits on a managed host, reverted on the next run.
The pattern: if a tool generates state from a definition, changing the state without changing the definition is temporary. The tool will regenerate from its source eventually, and it will not warn you that it is discarding your fix.
The corollary is more useful than the rule: if a fix is not in version control, it did not happen. My compose files are in Git. The February change was not, because it was never in a file. That absence is exactly why it disappeared and why nothing caught it.
What I Changed in How I Work
Fixes go in the declaration first. Change the compose file, redeploy from it, verify. Never docker update as the fix, only as an emergency stopgap explicitly followed by the real change.
Documented remediations get periodically re-verified. February’s writeup said the hazard was remediated. That was true when written and false six months later, and the documentation had no way to know. Anything documented as fixed and load-bearing needs a periodic check that it is still true.
Config diffs against the source during maintenance. Comparing what a container is actually running against what its compose file declares surfaces exactly this class of drift, and it takes seconds.
Takeaways
Runtime changes to declaratively managed resources are temporary. The definition wins eventually.
Recreation happens more often than you think. Any redeploy of any service in the same file regenerates the container.
A fix outside version control is not a fix. If it is not in a file that is tracked, it will vanish.
Verify running state against declared state after redeploying. Do not assume convergence, check it.
Re-verify documented remediations. Documentation records what was true once, not what is true now.
A hazard with no recent incidents is not a resolved hazard. Six quiet months meant the conditions had not aligned, not that the risk was gone.
Related: GPU Passthrough on Proxmox | Patching Seventeen Guests | Two Containers, One Missing Network