The Skills That Were Never Actually Written

Why this matters beyond the homelab: This is a sequel. An earlier post covered Jocasta fabricating tool call outputs mid-loop under load, describing a file operation instead of performing one. That was a live-loop failure, caught in real time with find and diff. This is the same failure mode discovered after the fact, sitting undetected in what looked like a completed body of work.

The reasoning leak, first

Before the audit, there was a smaller fix. Jocasta’s dashboard was leaking its internal reasoning into responses that should have been clean output. The fix should have been a single config key, and it took three wrong guesses before landing on the right one: display.show_reasoning. Confirmed live via a Discord test once it was actually the correct key. Not the interesting part of this post, but worth noting that even a config fix on this system doesn’t always go where the naming suggests it should.

Running hermes doctor afterward as a general health check turned up a “no API key” warning that looked alarming and turned out to be a false positive: it’s a check written for hosted API deployments, and it doesn’t know how to interpret a local Ollama backend correctly. Worth remembering the diagnostic tool has blind spots of its own.

The audit

The real work was a full knowledge-source audit, checking that everything Jocasta claims to know actually exists somewhere on disk. AnythingLLM checked out clean: 2,385 vectors confirmed intact in the Microsoft Learning workspace. The Postgres schema on Home One that was supposedly backing part of the knowledge base didn’t exist. Neither did the n8n workflows it was supposed to reference. Outline was empty.

And seven certification study-guide skills, each with a SKILL.md file describing content and referencing source files that were never actually written to disk. The session logs from when those skills were supposedly created show explicit “saved successfully” confirmations. The files simply aren’t there.

Same pattern, different surface

The earlier fabrication post was about a live tool call: the model described writing a file instead of writing it, caught within the same session. This is the same substitution, just discovered weeks later instead of in real time. The model reported success, the log recorded that success, and the actual artifact never existed. Nothing about the fabrication was different. What changed is how long it took to notice.

The lesson

A “saved successfully” message from an agent is a claim, not a receipt. The first fabrication post’s fix was procedural: discipline rules in SOUL.md, deterministic bash for file operations instead of trusting the model’s own report. This audit is the argument for why that fix needs to be permanent rather than a one-time patch. Anything an agent claims to have created is unverified until something outside the agent confirms it exists. That’s true the moment it’s created, and it’s still true months later when nobody’s checked.