k8s-monitoring Runbook

Metadata

Field	Value
Service	k8s-monitoring
Criticality	Tier 1
Owner	Platform / Observability owner
Namespace	k8s-monitoring
Clusters	homelab, local, jls
Last validated	2026-09-17
Related service page	../services/k8s-monitoring.md

Trigger Conditions

• Metrics, logs, or traces disappear from the shared observability backends.
• Alloy collectors restart repeatedly or accumulate remote write errors.
• OTLP, Jaeger, or Zipkin receiver endpoints stop accepting traffic.
• Helm or overlay upgrades change chart behavior between clusters.

1. Health Checks

Use these commands first to establish scope.

kubectl -n k8s-monitoring get deploy,statefulset,daemonset,pod,svc,pvc
kubectl -n k8s-monitoring get pods -o wide
kubectl -n k8s-monitoring get events --sort-by='.lastTimestamp' | tail -n 20
kubectl -n k8s-monitoring logs pod/<pod-name> --all-containers --tail=200

Probe verification

Collector workloads are rendered by the Helm chart, so verify readiness on the concrete Alloy pod showing trouble.

kubectl -n k8s-monitoring describe pod <alloy-pod-name>
kubectl -n k8s-monitoring get svc k8s-monitoring-alloy-receiver

Record:

• whether the failing component is alloy-metrics, alloy-logs, alloy-singleton, or alloy-receiver
• whether the failure is storage, auth, destination reachability, or receiver ingress related
• whether only one cluster overlay is affected because of version skew

2. Troubleshooting Workflows

Remote write or push failures to Prometheus or Loki

kubectl -n k8s-monitoring logs pod/<alloy-pod-name> --all-containers --tail=400 | grep -Ei '401|403|429|5..|remote_write|loki'
kubectl -n k8s-monitoring get configmap -o name | grep alloy

Check:

• destination URL is still reachable
• authentication values are valid for Prometheus, Loki, and Tempo
• errors are backpressure related rather than auth related

Kubernetes dashboards are empty while the scrape is healthy

The tell-tale is up being 1 for the scrape while none of its series exist:

# scrape is healthy in every cluster ...
curl -su promuser:<pw> --data-urlencode \
  'query=count by (cluster) (up{job="integrations/kubernetes/kube-state-metrics"})' \
  https://prometheus.mutana.site/api/v1/query
# ... but the metrics the dashboards need are absent
curl -su promuser:<pw> --data-urlencode 'query=count(kube_pod_info)' \
  https://prometheus.mutana.site/api/v1/query

That combination means the samples are dropped after the scrape, in Alloy's keep-rule, not lost in transit. Read the rendered rule rather than the values file:

kubectl -n k8s-monitoring get cm k8s-monitoring-alloy-metrics \
  -o jsonpath='{.data.config\.alloy}' \
  | grep -A4 'prometheus.relabel "kube_state_metrics"'

A healthy regex lists the whole default allow list. If it reads up|scrape_samples_scraped|... and little else, then clusterMetrics.kube-state-metrics.metricsTuning.useDefaultAllowList is false in that cluster's values — see k8s-monitoring.md §4, kube-state-metrics tuning, and fix the overlay and the matching inline values in k8s-monitoring/fleet.yaml.

If instead a specific kube_*_labels family is the only one missing, the allow list is fine and the kube-state-metrics Deployment is not exporting those labels:

kubectl -n k8s-monitoring get deploy k8s-monitoring-kube-state-metrics \
  -o jsonpath='{.spec.template.spec.containers[0].args}' | tr ',' '\n' | grep labels-allowlist

The resource must be listed there (pods=[...], nodes=[...]). Since kube-state-metrics v2.9 a kube_<resource>_labels family still prints its # HELP line while emitting zero samples when its resource is absent from that argument, so the metric looks present and reads as empty.

Panels that stay empty after kube-state-metrics series return are a different fault: the mixin recording rules. Check them at the Prometheus that evaluates them, not at the dashboard — see prometheus.md §4, Recording rules.

SNMP devices stop reporting

The homelab overlay polls OPNsense (192.168.1.1) and the Netgear GS724T (192.168.1.3) from alloy-singleton. A missing device shows up as a gap in the SNMP dashboards rather than as a collector failure, so check the scrape path in order.

# Target health straight from Alloy (all five SNMP components must be healthy)
POD=$(kubectl -n k8s-monitoring get pods -l app.kubernetes.io/name=alloy-singleton -o jsonpath='{.items[0].metadata.name}')
kubectl -n k8s-monitoring port-forward "$POD" 18099:12345
curl -s localhost:18099/api/v0/web/components | python3 -m json.tool | grep -i snmp
curl -s localhost:18099/metrics | grep 'prometheus_scrape_targets_gauge{component_id="prometheus.scrape.integrations_snmp"'

# Ask the exporter itself for one device's metrics (this is what a scrape does)
curl -s --max-time 60 \
  'localhost:18099/api/v0/component/prometheus.exporter.snmp.integrations_snmp/metrics?target=192.168.1.1&module=system,if_mib,hrDevice,hrStorage&auth=v2c_readonly' \
  | grep -E '^(sysName|sysUpTime|ifOperStatus)'

The Alloy image ships no SNMP tools, so walk a device from a host on the LAN instead (snmpget -v2c -c "$COMMUNITY" 192.168.1.1 1.3.6.1.2.1.1.5.0); if that host also fails, the problem is the device rather than the collector.

Check, in order:

• the device answers UDP/161 on the LAN and its SNMP agent is enabled
• alloy-snmp-community exists in namespace k8s-monitoring with key community; if it is missing or its value is wrong, the collector logs the failure and the exporter returns no series
• the target name in target "<name>" produced the expected snmp_target/instance labels; hyphens are rejected by the Alloy parser, so use underscores
• the scrape is not timing out: walking a 24-port switch with per-interface tables is far heavier than an HTTP scrape, which is why this job uses a 60s interval and a 30s timeout

If the ConfigMap volume does not refresh after a values change, restart the collector: kubectl -n k8s-monitoring rollout restart daemonset/k8s-monitoring-alloy-singleton.

OPNsense firewall metrics are missing or partial

The firewall is scraped over three channels and they fail independently: integrations/opnsense (node_exporter, port 9100), integrations/opnsense/telegraf (telegraf, port 9273) and integrations/snmp. A dashboard showing host metrics but no firewall or DNS figures means only the telegraf channel is broken.

# Which channels are up? Each failing one shows up as its own down target.
https://prometheus.mutana.site -> up{instance="opnsense"}

# Ask telegraf directly from any LAN host (this is what a scrape does)
curl -s http://192.168.1.1:9273/metrics | grep -E '^(pf|unbound|ping)_' | head

Check, in order:

• Connection refused on 9273 — the plugin is stopped or not installed: POST /api/telegraf/service/status should report running, and core/firmware/info should list os-telegraf as installed. Start it with POST /api/telegraf/service/reconfigure.
• /metrics answers but has no pf_* family at all — this is the signature of telegraf running as its default unprivileged user, which cannot execute pfctl or read /dev/pf. Set wheelgroup on the plugin's general tab (it renders telegraf_user="root" in the rc script) and reconfigure. Do not go looking for a Prometheus problem: the scrape is healthy and simply receives no pf series, so up stays 1 while the panels go empty.
• No unbound_* — the plugin's setup script adds telegraf to the proxy and unbound groups, so this usually means Unbound's remote control is off rather than a telegraf fault. Confirm with unbound-control stats on the firewall.
• No ping_* — the ping input targets 192.168.0.1 (ISP nexthop) and 1.1.1.1. If the ISP renumbers the WAN segment, update ping_hosts in the plugin's input tab and nothing in Git — the targets are configured on the firewall side, unlike the scrape target itself.
• The whole collector stopped after a config change — a bad Alloy block kills every scrape in the singleton, and a missing alloy-snmp-community Secret does the same. See the next section.

Editing the OPNsense side is a plugin UI/API change plus telegraf/service/reconfigure. Editing the cluster side is a values edit in overlays/homelab/values.yaml and its mirror in fleet.yaml, then a push.

Home Assistant stops reporting

The homelab overlay scrapes http://192.168.1.6:8123/api/prometheus as integrations/hass. Home Assistant answers 401 for a bad or revoked token and 404 when its Prometheus integration is not enabled — the two look alike in a dashboard gap but are different fixes.

G=https://grafana.mutana.fr   # or any host on the LAN
TOKEN=$(kubectl -n k8s-monitoring get secret alloy-hass-token -o jsonpath='{.data.token}' | base64 -d)
curl -s -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $TOKEN" \
  http://192.168.1.6:8123/api/prometheus

Check, in order:

• 404 means the Prometheus integration is off: add prometheus: to Home Assistant's configuration.yaml and restart Home Assistant. Nothing on the cluster side can enable it.
• 401 means the token is wrong or revoked. Long-lived tokens survive Home Assistant restarts but not a restore from backup; create a new one under the user profile and update the Secret: kubectl -n k8s-monitoring create secret generic alloy-hass-token --from-literal=token=<new> --dry-run=client -o yaml | kubectl apply -f -, then restart the collector (Alloy reads the Secret at evaluation, not on file watch).
• 200 from a LAN host but no metrics in the backend points at the destination, not at Home Assistant — check the remote write path instead.
• Home Assistant is plain HTTP on 8123; if scheme is ever set to https the scrape fails to connect rather than returning a status code.

The whole collector stops after a config change

If alloy-singleton restarts once and stays down, look for could not perform the initial load successfully in its log. Alloy aborts the entire initial load when any component fails to evaluate, and a remote.kubernetes.secret whose Secret does not exist does exactly that — so a config block referencing a missing Secret takes down every scrape in the collector, not just its own.

kubectl -n k8s-monitoring logs daemonset/k8s-monitoring-alloy-singleton -c alloy --tail=200 | grep -iE 'initial load|failed to evaluate'
kubectl -n k8s-monitoring get secret alloy-snmp-community alloy-hass-token alloy-awx-bearer-token

Create the missing Secret and restart. When adding a new scrape that authenticates from a Secret, create the Secret before deploying the block.

WAL or log-position storage failures

kubectl -n k8s-monitoring get pvc
kubectl -n k8s-monitoring describe pvc <pvc-name>
kubectl -n k8s-monitoring describe pod <alloy-pod-name>

Check:

• storageClass is present on the cluster
• hostPath /var/alloy-log-storage exists and is writable on the nodes running alloy-logs or alloy-singleton DaemonSet pods
• WAL volume is not full or stuck Pending
• for local cluster, confirm that the alloy-metrics StatefulSet PVC is node-affined to the correct node via volume.kubernetes.io/selected-node or by local-path-provisioner auto-affinity

Receiver endpoint is exposed but clients cannot connect

kubectl -n k8s-monitoring get ingressroute,middleware,secret
kubectl -n traefik logs deploy/traefik --tail=200 | grep -Ei 'otlp|jaeger|zipkin|k8s-monitoring'
kubectl -n k8s-monitoring get svc k8s-monitoring-alloy-receiver -o yaml

Check:

• receiver is enabled in the active overlay
• alloy-receivers-ingressroute.yaml is actually included in the kustomization
• authsecret still exists and the Traefik middleware chain is valid

Overlay drift or cluster-specific render problem

kubectl kustomize k8s-monitoring/overlays/<cluster> --enable-helm > /tmp/k8s-monitoring-rendered.yaml
grep -n 'alloy-receiver' /tmp/k8s-monitoring-rendered.yaml | head

Check:

• chart version and values shape are still compatible
• the intended collector set actually renders for the target cluster
• for the local overlay, verify no architecture-specific nodeSelector blocks DaemonSet pods on mixed-arch nodes

3. Disaster Recovery

Preconditions

• Confirm whether the priority is restoring telemetry ingestion or preserving short-lived collector state.
• Identify the active overlay and chart version for the affected cluster.
• Verify destination credentials and receiver basic auth secrets are available.

Stateful workload recovery

Most k8s-monitoring state is disposable. Recovery sequence:

1. Reapply the affected overlay with Helm enabled.
2. Recreate receiver auth secrets or ingress resources if exposure is required.
3. Restore or recreate PVCs and hostPath paths only if WAL continuity is needed.
4. Validate remote write, log push, and receiver health.

Cluster rebuild dependency order

1. Storage classes and node hostPath prerequisites
2. Traefik if receiver endpoints must be exposed externally
3. Alloy operator CRD registration
4. Helm-rendered k8s-monitoring workloads
5. Remote write and tracing validation

4. Scaling and Resource Management

Preferred path: adjust the overlay values.yaml and redeploy through GitOps.

Use these commands to size the problem before changing resources:

kubectl -n k8s-monitoring top pod
kubectl -n k8s-monitoring get statefulset,daemonset,deploy
kubectl -n k8s-monitoring describe pod <alloy-pod-name>

Record:

• which collector is saturated
• whether receiver load is driving memory pressure on alloy-receiver
• whether the chart currently uses PVC-backed or hostPath-backed local state

Current guidance: scale collector resources through values.yaml and avoid ad hoc live patches that are not reconciled back into the overlay.

5. Maintenance Procedures

• Review destination credentials and move them out of inline values where possible.
• Upgrade chart versions cluster by cluster and compare rendered workloads.
• Validate receiver exposure and auth middleware after Traefik changes.
• Periodically clear or resize WAL storage only with an explicit telemetry-loss decision.

For each task, define:

• Preconditions: target cluster overlay identified and destination credentials available
• Impact window: temporary telemetry loss or receiver unavailability
• Rollback path: restore previous values.yaml and kustomization.yaml revision
• Validation steps: confirm logs, metrics, and traces are flowing again

6. Rollback Strategy

Document the fastest safe rollback path:

• Revert the affected values.yaml or kustomization.yaml revision.
• Restore the previous receiver ingress resources if exposure changed.
• If WAL corruption is suspected, accept telemetry loss and recreate local state after confirming the service can resume cleanly.

7. Post-Incident Actions

After recovery, always:

1. Update CHANGELOG.md if credentials, receiver exposure, or chart versions were changed during the incident.
2. Update the k8s-monitoring service page if overlay versions, receiver status, or backend destinations changed.
3. Update this runbook if the incident exposed missing render, storage, or auth diagnostics.
4. Capture follow-up work to externalize inline credentials and confirm the local overlay's node placement for the alloy-metrics StatefulSet.