← Holocron Logs

The :latest Tag Lied: Fifteen Months Behind on Authentik Without Knowing It

Authentik was pulling :latest on every update and had been for over a year. It was running 2025.2.4. No error, no warning, no failed pull, just a tag that quietly refused to move because the project does not support skipping major versions. This is how I found it, why :latest cannot be trusted as an upgrade strategy, and the sequential hop plan to get current without destroying the identity provider that fronts fifteen services.

Why this matters beyond the homelab: Version drift on an identity provider is a security problem, not a housekeeping one. Every release you skip is a batch of authentication fixes you do not have. Worse, a mutable tag that silently stops advancing produces an environment that reports as current while being over a year stale, which is precisely the kind of gap that survives an audit.


The Discovery

I was patching application-layer services across the fleet, working through a checklist of seventeen guests. Authentik came up. It fronts SSO for fifteen-plus services via OIDC and SAML, so it gets checked carefully.

The compose file specified :latest. The nightly pull had been running for months. Everything reported healthy.

Then I actually looked at the running version:

docker exec -it authentik-server ak version

2025.2.4.

The current release was in the 2026.x line. I was more than a year behind on the single service that controls authentication to everything else, and every docker compose pull for that entire period had come back with nothing to do.


Why the Tag Stopped Moving

This is the part worth understanding, because it is not a Docker bug and it is not a broken registry.

Authentik does not support skipping major versions. Upgrades are expected to move through the release line in order, because each release ships database migrations that assume the schema left behind by the one before it. Jumping several majors at once means running migrations against a schema they were never written for.

The project’s tagging reflects that constraint. The :latest tag is not a promise to always serve the newest build to every possible starting point. Pulling it from a sufficiently old installation does not vault you forward, and critically, it does not error either. It simply resolves to something your instance can accept, and everything appears to succeed.

So the pull ran nightly. It exited zero. The container restarted healthy. And the version did not move.

That is the trap: the failure mode of :latest here is silence. There was nothing in any log to notice, no failed job, no alert. The only way to catch it is to ask the running application directly what version it is, which is not a thing most people think to do about a service that has been up for a year.


The Broader Lesson About Mutable Tags

:latest is convenient exactly until it is wrong, and when it is wrong it does not tell you.

Concrete problems with it on anything stateful:

You cannot describe your own environment. “We run Authentik latest” is not a version. It is not reproducible, it does not appear in your documentation meaningfully, and it cannot be diffed against a CVE advisory.

You cannot roll back deterministically. When an upgrade breaks something, the recovery move is redeploying the previous version. With a mutable tag you may not know what that was.

Upgrades happen on the registry’s schedule, not yours. For a stateless service that is fine. For an identity provider carrying its own database schema, an unplanned migration during a routine pull is a bad night.

And, as here, it can silently stop advancing without any signal that it has.

For anything holding persistent state, especially anything running schema migrations on startup, pin the explicit version tag and treat upgrades as a deliberate, planned operation.


The Upgrade Plan

Getting from 2025.2 to current cannot be one jump. The path is sequential, one major at a time:

2025.2  →  2025.4  →  2025.6  →  2026.1  →  2026.3  →  2026.5

Six hops. Every one gets the same treatment.

Before anything, a full database backup. Authentik’s state lives in Postgres and every hop runs migrations against it. A migration failure partway through leaves a schema that neither the old nor the new version fully understands.

docker exec -t authentik-postgres pg_dump -U authentik authentik \
  > /backup/authentik-preupgrade-$(date +%Y%m%d-%H%M).sql

Take a VM-level snapshot as well. The database dump covers the data, the snapshot covers everything else.

Each hop, explicitly:

services:
  server:
    image: ghcr.io/goauthentik/server:2025.4.4   # explicit, never :latest

Then, per hop:

  1. docker compose pull && docker compose up -d
  2. Watch the server logs until migrations complete and settle
  3. Confirm the version actually changed with ak version
  4. Log in through the UI
  5. Test an actual OIDC flow and an actual SAML flow against a real downstream service
  6. Confirm the worker process is running and processing tasks
  7. Only then take the next hop

The verification step people skip is the worker. Authentik’s worker handles background tasks, and it can fail independently of the server while the login page continues to look completely fine. A green login screen is not proof the upgrade succeeded.

Do not batch the hops. Six upgrades in one session means that when something breaks on hop four, you are debugging a schema that has been through four sets of migrations since your last known-good state. One at a time, verified, is slower and dramatically easier to recover from.


What I Changed Beyond This Service

The obvious fix is pinning Authentik. The useful fix is asking what else was lying.

I went through every compose file in the fleet and separated services into two groups:

Pinned to explicit versions: anything with a database, anything running migrations, anything in the authentication path. Authentik, the Postgres instances, Wazuh’s components, Gitea, BookStack, NetBox.

Left on floating tags: genuinely stateless things where an unattended jump is not a schema risk.

And a new habit: during any patch pass, ask the application its version, not the registry. docker compose pull reporting “up to date” means the tag did not move. It does not mean you are current. Those are different claims and only one of them matters.


Takeaways

Mutable tags fail silently. A successful pull is not evidence of a successful upgrade.

Version-check from inside the application. The registry’s opinion is not the same as what is running.

Pin anything stateful. Databases and migrations do not tolerate surprise version jumps, and identity providers are the last place you want one.

Sequential upgrades are not optional when a project says so. The constraint exists because the migrations are written that way.

Back up the database before every hop, not just the first. Each migration is a separate opportunity to need it.

Verify the worker, not just the login page. Background processing fails independently of the UI.


Related: Zero-Trust Identity Platform | Deploying Authentik SSO Across 15+ Services | Writing ADRs for a Homelab

← Back to Holocron Logs