Why this matters beyond the homelab: Backup coverage is the single most common gap between what an environment is documented to do and what it actually does. Jobs get defined once, new workloads get added later, and nobody reconciles the two. Auditing coverage against a live inventory rather than against your own assumptions is AZ-104 territory and the difference between a recovery plan and a hope.
The Trigger
A storage fault took down the VM holding BookStack, Gitea, and NetBox. Full recovery, no data lost, documented separately. But during that recovery I discovered the VM had never been backed up. Not once since it was created.
I have a nightly backup schedule. I have an offsite pipeline to Azure. Neither included the guest holding my wiki, my Git remote, and my network source of truth, because it was created after the jobs were defined and nobody went back to add it.
That is not a gap in design. That is a gap in verification, which is worse, because the design looked complete on paper and I believed it.
So I rebuilt the posture from the inventory up.
Layer One: Local Snapshots
Proxmox’s vzdump handles VM and container backups natively, and the important decisions are mode, compression, and retention.
vzdump 500 --mode snapshot --compress zstd --storage local-backup
Mode. snapshot uses the storage layer’s snapshot capability so the guest keeps running. stop is cleaner but means downtime. suspend is a compromise nobody actually wants. For anything with a database, snapshot mode plus a separate application-consistent dump (below) is the practical answer.
Compression. zstd is meaningfully faster than gzip at comparable ratios and is what I use everywhere.
Retention is configured per job rather than per invocation, and the trap is that retention is not a backup strategy by itself. Seven daily copies of a corrupted database is seven copies of a corrupted database. Retention depth matters most when the failure is silent, because you need to reach back past the point where corruption started.
The critical part is job membership. Backup jobs in the Proxmox UI select specific guests, and a guest created after the job exists is not in it. This is the entire reason my Codex VM was uncovered. Either select all guests explicitly, or make adding a guest to the backup job part of your provisioning checklist. Assuming it happens automatically is how you end up where I was.
Layer Two: Application-Consistent Database Dumps
A VM snapshot of a running Postgres instance is crash-consistent, not application-consistent. Postgres will usually recover from it the way it recovers from a power loss, which usually works. “Usually” is doing real work in that sentence.
So databases get dumped separately, on cron, before the VM backup window:
0 2 * * * docker exec -t postgres pg_dump -U authentik authentik \
| zstd > /backup/pg/authentik-$(date +\%Y\%m\%d).sql.zst
This runs for every Postgres instance in the fleet. The dumps land on disk and then get swept up by the VM backup and the offsite sync, so they inherit both layers of protection.
Ordering matters: dump first, then snapshot. That way the VM backup contains a fresh logical dump inside it, and you have two independent recovery paths from one backup, restore the whole VM, or pull just the SQL out of it.
Layer Three: Offsite to Azure
Local backups cover disk failure and mistakes. They do not cover the site, meaning fire, theft, or a mistake I make with sufficient enthusiasm.
Offsite runs to Azure Blob Storage via azcopy:
azcopy sync '/backup/' \
'https://<account>.blob.core.windows.net/<container>?<sas-token>' \
--recursive --delete-destination=false
The pieces that matter, and each of these maps to an AZ-104 objective:
sync rather than copy. Sync compares source and destination and transfers only differences. On a backup set that is mostly unchanged night to night, this is the difference between a manageable transfer and saturating the upload link.
--delete-destination=false. Deliberate. If something deletes my local backups, whether that is ransomware, a fat-fingered rm, or a failing disk, I do not want that deletion propagating to the offsite copy. The tradeoff is that the container grows and needs lifecycle management, which is the next point.
Storage tier and lifecycle policy. Backups are written once and read approximately never. A lifecycle rule moves blobs to Cool after 30 days and Archive after 90, then expires them, which is a large cost difference on data that exists purely as insurance. This is one of the clearest practical demonstrations of Azure storage tiering there is.
SAS token scoping. The token is write-scoped, container-scoped, and has an expiry date I have recorded, because a SAS token silently expiring is a backup pipeline silently stopping. Access keys would be simpler and would also mean full account access sitting in a cron job on a homelab box.
Layer Four: Verifying the Segment
The bot network on VLAN 50 had firewall rules written and never validated against live traffic. Rules that exist in a config are not rules that work, in exactly the same way that a backup job that exists is not a backup that ran.
Validation meant generating actual traffic for each rule, allowed and denied, and confirming from the counters that traffic hit the rule I expected rather than falling through to a default. Several rules were being shadowed by earlier, broader rules, which is invisible until you check hit counts rather than reading the config top to bottom.
Layer Five: The Token Audit
The closeout was auditing every API token the observability bot holds, across Proxmox, Wazuh, InfluxDB, and Uptime Kuma.
What I was checking:
Scope. Every token read-only, and every one actually verified read-only by trying a write and confirming it was refused. A token documented as read-only and a token that is read-only are different claims.
Validity. Tokens expire and get regenerated, and the failure mode is a monitoring bot going quiet rather than erroring loudly. A 401 from a monitoring integration means you have no monitoring, which is worse than an alarm.
Storage. Secrets in the vault, referenced by env file, never pasted into a compose file that ends up in Git.
Separation. Dedicated service accounts per bot rather than tokens minted under my admin account, so audit logs attribute actions to the thing that took them.
A monitoring system with expired credentials reports nothing and looks identical to a monitoring system reporting that nothing is wrong. That distinction is the whole reason this audit exists.
Takeaways
Audit coverage against a live inventory. Enumerate every guest, check each against the backup job. Do not audit against your own memory of what the jobs cover.
New workloads do not join backup jobs by themselves. Make it a provisioning step or it will not happen.
Databases need logical dumps, not just VM snapshots. Dump first, snapshot second, get two recovery paths from one run.
Do not propagate deletions offsite. The offsite copy exists to survive things that destroy the local one, including deliberate deletion.
Lifecycle policies are how offsite backup stays affordable. Write-once, read-never data belongs in cold tiers.
Track SAS expiry as an operational deadline. An expired token is a silently stopped pipeline.
Verify read-only tokens by attempting a write. Documented scope and actual scope diverge.
A restore you have never tested is a theory. Everything above is coverage, not proof. Testing an actual restore is the next post.
Related: The Alliance Fleet Backup Architecture | The Disk That Only Failed Halfway | The Network Layer