Inference worker

The inference worker runs on a separate Bazzite workstation rather than the Rocky Linux and K3s host. Rootless Podman and a systemd Quadlet manage a pinned CUDA-enabled llama.cpp server image. A checksum-pinned Qwen3.8-27B IQ4-class GGUF is mounted read-only.

The API binds to a stable private address, requires an API key, and has neither a router forward nor a public Caddy route. The key authenticates requests but does not encrypt plain HTTP, so that transport is acceptable only on the trusted LAN or WireGuard path.

Separating inference keeps GPU drivers, large model files, and desktop resource pressure away from the stable server. It also makes classification availability depend on a workstation that can be unavailable for gaming or maintenance.

Container and GPU boundary

NVIDIA CDI supplies the GPU to Podman. The persistent service initially loaded its model and listened successfully while startup logs showed CUDA unavailable and GPU layers ignored. Health checks therefore passed even though inference had fallen back to the CPU.

Adding SecurityLabelDisable=true, matching NVIDIA's Podman CDI requirement, resolved GPU handoff. That disables SELinux separation for this container, so the compensating boundaries are narrow: the image is pinned, the model mount is read-only, unrelated host paths are not mounted, and the API stays private.

Validation evidence

The initial profile used an NVIDIA RTX 5070 Ti with 16 GB of memory, an 8,192-token context, one inference slot, and text-only operation. Local health and authenticated chat completion passed. The server process was observed using 12,522 MiB of GPU memory, and the private health endpoint was reachable from both Rocky Linux and a Linkwarden pod.

Those measurements establish cross-host reachability and GPU execution; they do not establish latency, throughput, repeated-request stability, or high availability. Validation therefore treats driver state, CDI discovery, disposable-container GPU visibility, model checksum, generated Quadlet, service state, GPU memory use, health, and authenticated completion as distinct layers. An active service is insufficient when its logs show CPU fallback.

A separate reboot failure traced a bind error to a DHCP reservation associated with an old or changed MAC address. The diagnostic boundary compares the active interface MAC, reserved private address, and configured bind address before changing the service.