Why this matters beyond the homelab: Self-hosting everything is a reasonable default in a homelab built partly to learn infrastructure. It’s not a rule that has to apply to every single service, and K-2SO was the case where it clearly shouldn’t.
The job K-2SO actually does
K-2SO is a small discord.js bot whose entire purpose is answering infrastructure questions in the fleet’s Discord server: status checks, quick lookups, the kind of thing that needs to be fast and reliably correct far more than it needs to demonstrate a self-hosted model pipeline. It was previously running against a locally hosted model, sharing GPU and inference resources with the rest of the fleet’s AI workloads on Tantive-III.
The tradeoff
Running K-2SO locally meant its response time was tied to whatever else was competing for that GPU at the time, and it added one more consumer to hardware that’s already carrying Jocasta’s inference load. None of that is free, and none of it was buying K-2SO anything it actually needed. It doesn’t need a fine-tuned local model or the privacy properties of keeping every query on-box. It needs to answer a Discord message quickly and correctly.
Moving it to claude-haiku-4-5 over the Anthropic API removed the local dependency entirely and brought response time down to under three seconds, consistently, regardless of what else is running on Tantive-III at the time.
The lesson
Self-hosting is worth it when the workload benefits from it: privacy, cost at scale, learning the infrastructure itself, control over the model. A lightweight utility bot answering status queries benefits from none of those things, it benefits from being fast and not contending for GPU time with everything else in the fleet. Not every piece of the stack needs to make the same architectural argument. Knowing which ones don’t is as much a part of the design as knowing which ones do.