Why this matters beyond the homelab: Patching is where most environments accumulate risk, either by not doing it or by doing it without a rollback path. A repeatable method with a snapshot before each change and a verified baseline after is the difference between routine maintenance and an unplanned outage with no way back.
The Method
Seventeen guests: ten LXCs and service stacks, six VMs. The procedure was identical for every one, and the discipline of not varying it is most of the value.
1. Establish a baseline before touching anything. Not “it seems fine.” A recorded list of which services should be running, which ports should answer, and what the version numbers currently are. You cannot verify a successful patch without knowing what success looked like beforehand.
2. Snapshot. Every guest, every time, before any change.
vzdump 500 --mode snapshot --compress zstd --storage local-backup
The snapshot that turns out to be unnecessary costs a few minutes. The one you needed and did not take costs a rebuild.
3. Patch.
4. Verify against the baseline from step one. Every service back up, every port answering, versions actually changed.
5. Record what happened. Version deltas and anything surprising, written down at the time rather than reconstructed later.
Boring by design. The findings below are what made it worth doing.
Finding: Keep Your Config, Not the Maintainer’s
The reverse proxy container upgraded openresty from 1.27 to 1.31, and mid-upgrade apt asked the question that deserves real attention:
Configuration file '/etc/nginx/nginx.conf'
==> Modified (by you or by a script) since installation.
==> Package distributor has shipped an updated version.
What would you like to do about it?
The default is to keep your existing version, and that was correct here. My nginx.conf carries proxy configuration for every service behind it. Accepting the maintainer’s version would have replaced it with a stock config and taken down external access to the entire fleet.
Two rules worth internalizing:
Never accept a config replacement reflexively. Diff first (D at that prompt), understand what changes, then decide.
Know which files you have modified before you start patching. If you cannot answer “have I customized this?” instantly, you are guessing under time pressure. This is the strongest argument for keeping service configs in version control rather than only on the box.
Finding: Agents Held Back With Missing Unit Files
Two nodes had the monitoring agent in a strange state. It was held back by apt, and its systemd unit file was missing entirely.
The hold meant apt upgrade skipped it silently, which is what a hold is supposed to do, so it had been sitting stale without complaint. The missing unit meant systemd could not manage it at all.
apt-mark showhold
apt install --only-upgrade wazuh-agent
--only-upgrade upgrades the package if installed without pulling it in fresh where it is not, which is the right verb when repairing a partial install.
Here the correct answer on the config prompt was the opposite of the openresty case: accept the package maintainer’s systemd unit file, because I had not customized it and the local state was the broken thing. Same prompt, opposite decision, and the deciding factor is whether the local version represents intentional customization or accumulated damage.
The broader lesson is that apt-mark showhold belongs in your patching checklist. Held packages are invisible during upgrades by design. A package held six months ago for a good reason that no longer applies is a permanently stale component nobody is looking at.
Finding: A False Outage From a Stale Orchestrator Entry
The container orchestrator showed one node as Down. The node was fine. Every container on it was running and reachable.
The orchestrator was holding a stale IP for that node’s agent. The address had changed at some point and the environment entry had not been updated, so the orchestrator was polling nothing and reporting the node dead.
Upgrading the orchestrator and correcting the entry cleared it.
The reason this matters more than a cosmetic bug: a monitoring system reporting a false outage trains you to ignore it. Once you learn that a red indicator means “check whether the monitoring is wrong,” the indicator has stopped conveying information. False positives degrade a monitoring system exactly as much as missed detections do, just more slowly.
Finding: A Fix That Reverted Because I Fixed the Wrong Layer
The AI node runs both an inference server and an image generation service, both wanting the GPU. Back in February, simultaneous load from both locked up the node hard. I fixed it at the time and documented it.
During this pass I found the image generation service running with restart: unless-stopped, exactly the configuration that caused the February incident. It would restart automatically after any stop, re-entering contention for the GPU without me involved.
The February fix had been applied to the running container, not to the compose file that defines it. The next docker compose up -d regenerated the container from the unchanged source and silently reverted the fix. It had probably been reverted for months.
Fix the declaration, not the instance. Anything changed on a running container is temporary by definition, because the compose file is the source of truth and it will win eventually. Changes made only to running state have an expiry date you will not be notified about.
Corrected in the compose file, redeployed, and only then did I proceed with the GPU toolkit upgrade, verifying passthrough survived two full Docker daemon restarts before considering it done.
Findings Worth Recording Even When Nothing Broke
Several components upgraded cleanly and still deserved verification beyond “the container started.”
The metrics database and dashboard stack both moved several versions. Both were verified end to end, meaning an actual dashboard rendering actual current data, not just two containers reporting healthy. A dashboard that loads while querying a database that lost its retention policy looks identical to a working one until you notice the graphs are empty past a certain date.
The SIEM’s manager, indexer, and dashboard were upgraded together, in one operation. Those three components negotiate compatibility with each other and a version skew between them produces failures that look like network or auth problems rather than version problems. Multi-component systems get upgraded as a unit.
Takeaways
Baseline first, or you cannot verify. Record what running looks like before you change anything.
Snapshot before every guest. Not before the risky ones. Before all of them.
Never answer a config-replacement prompt on reflex. Diff, then decide. The right answer differs case by case.
Check apt-mark showhold before patching. Held packages are silently stale by design.
A false outage costs you as much as a missed one. Fix stale monitoring entries promptly.
Fix the declaration, not the instance. Changes to running containers revert without warning.
Upgrade multi-component systems together. Version skew between coupled components produces confusing failures.
Verify end to end, not just process state. A healthy container is not a working service.
Related: One Weekend, Three Fires | Portainer: Container Orchestration | Writing ADRs for a Homelab