Why this matters beyond the homelab: IP conflict diagnosis, coordinated kernel upgrades across a cluster, and driver/kernel dependency management are bread-and-butter incidents in any production Linux environment. The tooling scales; the method is identical.
Fire One: Two Machines, One Address
The symptom was maddening in the specific way network-layer bugs are: intermittent. Connections to Coruscant, the VM hosting the fleet’s AI agent, worked, then stalled, then worked again. No service errors. No resource pressure. Just a host that was sometimes reachable and sometimes not, which is the signature of something below the application noticing.
The ARP table told the story. Two machines, Stinger Mantis and Coruscant, were both claiming .80 on the services VLAN. Every device on the segment was flip-flopping its ARP cache between two MAC addresses depending on which host had answered last. Traffic destined for one machine was intermittently delivered to the other, and neither host logged anything, because from each machine’s perspective nothing was wrong.
Duplicate IPs are almost always a provisioning bookkeeping failure, and this one was mine: two VMs configured at different times, each assigned an address that was “free” at the moment of assignment. The fix was mechanical (Coruscant moved to 192.168.20.82, ARP caches flushed, connectivity instantly stable). The prevention was procedural: IP assignments now live in NetBox as the authority, and an address is not free until the IPAM says it is free. Spreadsheet-memory IPAM fails exactly this way, in every environment, at every scale.
Fire Two: The Kernel Upgrade
With the network stable, all three Proxmox nodes took the upgrade to kernel 7.0.14-4-pve. Cluster kernel upgrades are a choreography problem: one node at a time, migrate workloads off, upgrade, reboot, verify quorum, take the workloads back, next node.
This round doubled as compliance work. ADR-008 defines VM placement policy across the fleet, and two machines (Home-One and Coruscant) had drifted onto the wrong node. The migration windows for the kernel upgrade were the natural moment to put them where the ADR says they belong, on Node-B. Two changes, one maintenance window, both documented before execution.
Nodes one and two came back clean. Node three was Tantive-III, the AI inference host.
Fire Three: The GPU That Wasn’t There
Tantive-III rebooted onto the new kernel and Ollama had no GPU. nvidia-smi returned the error every Linux GPU operator dreads: the driver could not communicate with the hardware.
The mechanism is worth understanding because it recurs on every kernel bump. NVIDIA’s driver is built per-kernel by DKMS. When a new kernel installs, DKMS rebuilds the module against that kernel’s headers. No headers for the new kernel, no rebuild, no module, no GPU. The failure is silent at upgrade time and loud at boot time.
Diagnosis confirmed it: the linux-headers package for 7.0.14-4-pve was absent, so the DKMS build had never run. Installing the headers and triggering the rebuild brought the module back, with one addition: nopvspin added to the GRUB command line to settle paravirtualized spinlock behavior under the new kernel.
apt install pve-headers-$(uname -r)
dkms autoinstall
# GRUB_CMDLINE_LINUX_DEFAULT += nopvspin
update-grub && reboot
Inference workloads confirmed back on GPU. The permanent fix is making headers a dependency of the upgrade runbook, not a thing remembered after the reboot fails.
Key Takeaways
- Intermittent connectivity with clean logs on both endpoints means look below the application. The ARP table is a five-second check that solves a category of “impossible” bugs.
- IPAM is not optional once you pass roughly a dozen hosts. The authority for address assignments has to be a system, not a memory.
- DKMS failures are predictable: driver modules are per-kernel artifacts, and headers are the build dependency. Put them in the upgrade runbook.
- Bundling ADR compliance into planned maintenance windows turns documentation debt into routine work instead of a standalone project.