Why this matters beyond the homelab: Most retry logic gets written for the happy path: the call fails once, it retries, it succeeds, nobody thinks about it again. The failure mode that actually matters is what happens when the call keeps failing and nothing tells the retry loop to slow down.
What happened
Jocasta posts to Discord as part of its normal operation, enriching alerts and responding through its webhook integration. One of those calls failed. Instead of backing off, the retry logic fired again immediately, and again, and again, with no increasing delay between attempts and no ceiling on how many times it would try.
Each retry meant re-running the work behind that call, which meant repeatedly hitting the GPU-backed inference backend on Tantive-III rather than a single lightweight network request quietly failing in the background. A tight loop with no backoff turned a transient Discord issue into sustained, repeated load on hardware that was never meant to be hammered that way, and it showed up as a thermal event on the card.
The actual gap
The retry logic itself wasn’t wrong to exist. Retrying a failed call is normal and usually correct. What was missing was the part that keeps a retry loop from becoming a stress test: exponential backoff, a maximum retry count, or both. Without either, “the call failed” and “the call is going to keep failing at full speed forever” look identical to the code, and the loop can’t tell the difference between a blip and a sustained outage.
The lesson
Any retry logic touching something expensive, GPU inference, an API with rate limits, anything with a real cost per attempt, needs an explicit ceiling on how aggressively it’s allowed to fail. A backoff isn’t just politeness toward the thing you’re calling, it’s a safety valve for your own hardware when the thing you’re calling doesn’t come back quickly. This is a small fix once it’s identified. The value here was recognizing that a Discord webhook failure and a GPU thermal alert were the same incident, not two separate problems that happened to show up on the same day.
Update, October 2026
The same failure class came back from a different direction. An n8n workflow that enriched Wazuh alerts sent one request per alert to Jocasta with no limit on how many could be in flight, and the resulting flood pinned the same GPU at 96% utilization. Different sender, same missing ceiling. The write-up is in The Alert Enrichment Workflow That Heated a GPU.