Maintenance

Lifecycle

The operating system package manager covers only part of the environment. Custom Caddy builds, K3s, Kubernetes workload images, classifier dependencies, model files, and the inference container each have their own lifecycle.

Each lifecycle begins with discovery, captures backup or recovery evidence where state can change, validates inputs before mutation, checks stability afterward, and retains a rollback path. Mutable image tags are avoided where recovery identity matters. An application image rollback is never assumed to reverse database or search-index changes, and recovery artifacts remain until the upgraded service has passed its observation period.

Bootstrap and maintenance are separate. Bootstrap creates an absent component and refuses to replace an existing installation silently. Maintenance starts from discovered live state and owns upgrades, migrations, rollback, and recovery evidence.

Troubleshooting

Diagnosis proceeds from the outside inward and stops at the first failed boundary:

1. DNS and router forwarding.
2. Host firewall and listening socket.
3. Caddy configuration, service, certificate, and upstream reachability.
4. K3s node, namespace, Service, Deployment, pod, and events.
5. PVC binding and filesystem or SELinux state.
6. Application logs and dependency health.
7. Public HTTP response and an authenticated functional check.

This order prevents network, ingress, orchestration, storage, and application failures from being diagnosed as one undifferentiated outage. The same principle applies within components: CrowdSec detection is separated from firewall enforcement, and inference health is separated from actual GPU execution.