Why this matters beyond the homelab: A warm node reads as a hardware problem: airflow, dust, a fan curve. This one wasn’t. The heat was real, but it was being produced by software asking a GPU for work faster than the GPU could ever finish it. The same failure class had already shown up a month earlier, and the second occurrence is the one that makes it a pattern instead of an accident.
The symptom
Node-A (FCM2250) was running hot, and the GPU in Tantive-III was the source. It wasn’t idle. It was at 96% utilization, drawing 69W against a 70W cap, with the card at 91C. Ten parallel connections were hitting Ollama with back-to-back chat completion requests, and the prompt was the same roughly 5,200 tokens every time.
Finding the sender
Ollama only answers what it is sent, so the question was who was asking. The traffic came from the Hermes gateway on Coruscant, the same agent runtime behind Jocasta. That gateway talks to three things: Discord, Home Assistant, and a webhook endpoint. The gateway log made the source clear. Hundreds of parallel webhook:alert-enrichment sessions were arriving on the webhook port. Not Discord, not Home Assistant.
On the other side of that webhook was the n8n Command Bridge. One of its workflows enriches Wazuh alerts by posting each one to Jocasta for an AI summary, and it sent one request per alert. There was no batching, no deduplication, and no limit on how many could be in flight. A burst of alerts became a burst of inference jobs, and each one queued behind the last on a card that can only run so many at once. The gateway’s session database had grown to 162MB from the stale sessions.
Stopping it
Killing the gateway process did nothing useful, because the service is set to Restart=always. Systemd brought it straight back. The change that held was stopping and disabling it through systemd:
systemctl stop hermes-gateway
systemctl disable hermes-gateway
The GPU fell to 2W and 0% utilization immediately, and the node settled back toward its normal baseline in the high 60s.
That treated the symptom. The cause was the workflow, so the n8n alert webhook workflow was deactivated too, and the gateway was brought back afterward. Wazuh alerts still reach Discord, because the workflow posts to Discord webhooks directly by severity. Only the AI enrichment tail is off.
What has to exist before enrichment comes back
Three controls were missing on the sending side, and all three are needed:
- A concurrency cap, so no more than a few enrichment jobs are ever in flight.
- Batching or deduplication, so ten alerts about the same event become one request, not ten.
- A queue with backpressure, so a slow receiver slows the sender instead of piling up work.
A GPU power cap such as nvidia-smi -pl 45 was the quick answer on the table. It would have limited the heat, but it would also have hidden the problem: the card would have run cooler while still answering hundreds of redundant requests. Digging into the gateway log until the real sender turned up is what converted a thermal symptom into a fixable cause. A cap may still be worth having as a last-resort ceiling, but it is a backstop, not the fix.
The lesson
The September retry loop and this flood are the same bug wearing different clothes: a sender with no idea the thing it calls is expensive. A Discord call that retried without backoff and a workflow that fanned out without a limit both ended at the same GPU. The real fix belongs where the work is created, a limit on the sender, because a hardware cap only hides the next flood and the next sender to misbehave won’t announce itself either.
Related: A Missing Backoff Turned a Discord Hiccup Into a GPU Problem | Building a Command Bridge: Six n8n Workflows and the Bugs That Made Them Work | Down for Two Weeks and Nobody Noticed: The Restart Policy I Forgot | Beszel: Host-Level Monitoring Without Running Another Stack