k8s-monitoring

Metadata

Field	Value
Service	k8s-monitoring
Purpose	Collect, transform, and forward metrics, logs, events, and traces from cluster workloads to the shared observability backends
Criticality	Tier 1
Owner	Platform / Observability owner
Clusters	homelab, local, jls
Namespace	k8s-monitoring
Exposure	Internal by default, with external OTLP and tracing receiver endpoints on selected overlays
Stateful	yes
Backup class	Best-effort PVC and hostPath state, Git-backed configuration
RPO / RTO	Telemetry loss is acceptable during rebuild, 30 to 60 minutes to restore collection paths
Last reviewed	2026-09-17

1. Service Overview

k8s-monitoring is the shared observability pipeline built from the Grafana k8s-monitoring Helm chart plus the Alloy operator CRD. It collects cluster metrics, pod logs, events, and optional receiver traffic, then forwards telemetry to shared Prometheus, Loki, and Tempo destinations.

Summary

The service is not part of workload delivery, but it is essential for diagnosing incidents and maintaining platform visibility. The repository carries several overlay versions in parallel, and cluster-specific values determine whether Alloy receivers are enabled and externally exposed.

The homelab overlay also reaches outside the cluster to poll network hardware. Its alloy-singleton collector scrapes the OPNsense firewall over two HTTP endpoints — the os-node_exporter plugin on port 9100 and the os-telegraf plugin's Prometheus output on port 9273 — and polls the firewall and the Netgear GS724T switch over SNMP v2c, forwarding all of it to the same Prometheus destination as the in-cluster scrapes. That makes the homelab overlay depend on the two devices answering UDP/161 and on the firewall answering TCP/9100 and TCP/9273, and on the alloy-snmp-community Secret existing in namespace k8s-monitoring — the collector fails its config load if the Secret is absent.

Dependencies

Dependency	Type	Why it matters
Grafana k8s-monitoring Helm chart	packaging	The workload is rendered by Helm through Kustomize
Alloy operator CRD	operator dependency	Collector resources depend on the Alloy custom resources being registered
Prometheus and Loki endpoints on mutana.site	telemetry backend	Remote write and log push paths terminate there
Grafana Cloud Tempo	trace backend	OTLP traces are forwarded to the hosted Tempo endpoint
Network devices on the LAN (OPNsense, Netgear GS724T)	monitored hardware	The homelab alloy-singleton polls them over SNMP v2c on UDP/161 and needs the alloy-snmp-community Secret to authenticate
OPNsense os-node_exporter (192.168.1.1:9100)	monitored host	Scraped over HTTP as integrations/opnsense for host metrics
OPNsense os-telegraf (192.168.1.1:9273)	monitored host	Scraped over HTTP as integrations/opnsense/telegraf; the only source of pf firewall, Unbound resolver, and nexthop/upstream reachability metrics. Requires the os-telegraf plugin installed, enabled, and running as root (plugin general tab, wheelgroup), otherwise the scrape is healthy but emits no pf_* series
Home Assistant (192.168.1.6:8123)	monitored host	Scraped over its Prometheus integration with a bearer token from the alloy-hass-token Secret; the endpoint must be enabled on the Home Assistant side
Traefik and Authelia	ingress	Receiver-enabled overlays expose OTLP, Jaeger, and Zipkin endpoints through Traefik
Storage classes and hostPath mounts	state	Alloy WAL and log positions depend on PVCs and hostPath mounts

2. Architecture Diagram

[Cluster metrics, logs, events]
  -> [Alloy metrics / logs / singleton collectors]
  -> [Prometheus remote write and Loki push on mutana.site]

[External OTLP and tracing clients]
  -> [Traefik IngressRoute]
  -> [Alloy receiver service]
  -> [Grafana Cloud Tempo]

3. Deployment Specifications

Item	Value
Source path	k8s-monitoring/overlays/*
Deployment model	Dual-mode Helm: Fleet inflates the chart natively then applies overlays//fleet (Alloy CRD, namespace, receiver IngressRoute, destination credential secrets) as post-render; local rendering uses kustomize --enable-helm on the same values file
Namespace	k8s-monitoring
Workload kind	Helm-managed Deployments, DaemonSets, StatefulSets, Services, and Alloy custom resources
Chart or image version	grafana/k8s-monitoring chart 4.5.0 (bundles alloy-operator 0.6.3) — pinned in fleet.yaml and each overlays//kustomization.yaml (keep in sync)
Config files	overlays/*/values.yaml (single source of truth per cluster) plus the shared overlays//fleet/ layer (vendored Alloy CRD pinned to the bundled operator version)

Cluster mapping

Cluster	Overlay path	Notes
homelab	k8s-monitoring/overlays/homelab	Receiver exposed on home.mutana.fr hosts
local	k8s-monitoring/overlays/local	Mixed arm64 and amd64 nodes; alloy-metrics StatefulSet uses local-path WAL; alloy-logs and alloy-singleton run as DaemonSets on all nodes; alloy-receiver enabled and exposed via Traefik on oci-arm
jls	k8s-monitoring/overlays/jls	Ships metrics and logs to the self-hosted Prometheus and Loki only: no Grafana Cloud destination, no Fleet Management remote config, no receiver (all removed 2026-09-23); jelastic-dynamic-volume for Alloy WAL PVCs

4. Configuration Guide

The values files define telemetry destinations, collector presets, storage, and optional receiver exposure. Configuration differs materially between overlays.

Environment variables

Variable	Source	Purpose	Secret?
Generated by the Helm chart	chart values and templates	Collector runtime settings are rendered from values.yaml rather than declared manually in the repository	mixed

ConfigMaps

Resource	Path	Purpose
Chart-generated Alloy configuration	k8s-monitoring/overlays/*/values.yaml	Collector pipelines, custom scrape jobs, and destination wiring are rendered from Helm values

Secrets management

• Secret names: authsecret for receiver basic auth on overlays that include alloy-receivers-ingressroute.yaml; alloy-snmp-community (key community) for the SNMP v2c community used by the homelab overlay's network device polling
• Source of truth: the receiver auth secret is committed as Kubernetes Secret data inside alloy-receivers-ingressroute.yaml, while destination authentication is currently embedded inline in several values.yaml files. The SNMP community is created out of band and read by Alloy through remote.kubernetes.secret, so no community value is tracked in Git
• Rotation trigger: credential rotation for Prometheus, Loki, Grafana Cloud Tempo, or receiver basic auth
• Recovery note: treat the inline credential pattern as technical debt and migrate destination authentication to Secret references before broader rollout

Notable configuration facts from the current overlays:

• local uses local-path for alloy-metrics WAL PVC; alloy-logs and alloy-singleton use hostPath at /var/alloy-log-storage on each node.
• The local cluster is mixed-architecture (oci-arm and oci-arm-free1 are arm64; layer7-vps1 is amd64); no architecture-specific nodeSelector is set for DaemonSets or the receiver, so they run on all nodes.
• alloy-metrics is a StatefulSet and will pin to the node where its local-path PVC was first provisioned; review the PVC's selected-node annotation if the pod cannot reschedule.
• The ingress node for the local cluster is oci-arm (node.io/ingress=true, arm64); alloy-receiver is exposed through Traefik running on that node.
• homelab and local enable alloy-receiver and publish OTLP, Jaeger, and Zipkin ports. jls does not: its receiver carried traces only, and traces went only to the Grafana Cloud stack removed on 2026-09-23.
• homelab polls two SNMP devices from alloy-singleton: opnsense (192.168.1.1) and netgear_gs724t (192.168.1.3), using the modules system,if_mib,hrDevice,hrStorage at a 60s interval and 30s timeout rather than the cluster-wide 15s interval. The exporter contract is the Grafana Cloud SNMP integration one, so job_snmp, snmp_target, and instance are the labels the dashboards in grafana-dashboard/dashboards/SNMP select on.
• homelab scrapes the OPNsense firewall three ways and the split matters: integrations/opnsense (port 9100, os-node_exporter) carries host metrics, integrations/opnsense/telegraf (port 9273, os-telegraf) carries the OPNsense-native ones, and integrations/snmp carries interfaces and hardware. Both HTTP scrapes are pinned to instance=opnsense at the cluster-wide interval; the SNMP walk is not.
• The telegraf input list is deliberately narrow — pf, unbound, ping, system, processes. Its cpu, mem, disk, diskio, network, zfs and ntpq inputs are off because os-node_exporter already reports the same values from the same host, and intrusion_detection_alerts is off because it tags each Suricata event with src/dest IP and port, which is unbounded cardinality. See the k8s-monitoring README for what to do before enabling either.
• homelab also scrapes Home Assistant at 192.168.1.6:8123 over HTTP as integrations/hass, instance homeassistant, keeping a bounded set of entity metrics. It depends on a long-lived token in the alloy-hass-token Secret and on the prometheus integration being enabled on the Home Assistant host.
• Both scrapes authenticate from out-of-band Secrets, and here that dependency is load-bearing in an unusual way: a remote.kubernetes.secret Alloy cannot read aborts the collector's whole initial load, so a missing Secret stops every scrape in that collector rather than just the one that references it. Create the Secret before deploying a block that reads one.

kube-state-metrics tuning

Two keys look interchangeable and are not. They sit in different charts and are easy to swap by mistake:

Key	What it controls	Symptom when wrong
clusterMetrics.kube-state-metrics.metricsTuning	The Alloy prometheus.relabel keep-rule, i.e. which scraped series survive to remote write	Metrics are scraped (up is 1 for integrations/kubernetes/kube-state-metrics) but never arrive in Prometheus
telemetryServices.kube-state-metrics.metricLabelsAllowlist	The kube-state-metrics Deployment's --metric-labels-allowlist argument	kube_*_labels families are declared in # HELP but emit zero samples

metricLabelsAllowlist under clusterMetrics is silently ignored — the chart neither reads nor rejects it, and make check cannot catch it.

metricsTuning.useDefaultAllowList must stay true. The default list is what the vendored Kubernetes dashboards and the kubernetes.recording rules query (kube_pod_info, kube_pod_owner, kube_node_status_allocatable, kube_pod_container_resource_*, kube_pod_status_phase, kube_resourcequota). Setting it false reduces the keep-rule to whatever includeMetrics lists and silently empties every Kubernetes dashboard. includeMetrics adds to the default list rather than replacing it, so kube_pod_labels — which is not in the default list — is requested there, and its label_* dimensions come from the pods= entry of the telemetryServices allowlist.

To check what a cluster is actually keeping, read the rendered keep-rule rather than the values file:

kubectl -n k8s-monitoring get cm k8s-monitoring-alloy-metrics \
  -o jsonpath='{.data.config\.alloy}' \
  | grep -A4 'prometheus.relabel "kube_state_metrics"'

5. Access Protocols

Path	URL or endpoint	Audience	Auth	TLS terminates at
Internal	k8s-monitoring-alloy-receiver.k8s-monitoring.svc.cluster.local on ports 4318, 4317, 14250, 6832, 6831, 14268, and 9411 when receiver is enabled	Cluster workloads and operators	Service-specific or none inside the cluster	Service or ingress path dependent
External	otlp-http, otlp-grpc, jaeger-grpc, jaeger-binary, jaeger-compact, jaeger-http, and zipkin under home.mutana.fr (homelab) and mutana.fr (local)	Telemetry clients and operators	Traefik middleware chain plus basic auth middleware alloy-auth	Traefik

6. Operations and Observability

• Primary health indicators: collector pods Ready, remote write pipelines healthy, and no sustained backlog in Alloy WAL or log position storage.
• Dashboards or alerts: Prometheus remote write status, Loki push success, and Alloy pod logs.
• Log locations: alloy-metrics, alloy-logs, alloy-singleton, and alloy-receiver pod logs in k8s-monitoring.
• Known failure modes: invalid destination credentials, receiver ingress misconfiguration, local-path PVC pending because a node has been lost or a PV is node-affined to the wrong node, and overlay drift between chart versions.

7. Backup and Recovery Notes

• Backup method: configuration is Git-backed, but Alloy WAL and log-position state are best-effort and can be recreated with temporary telemetry loss.
• Restore prerequisites: namespace k8s-monitoring, Alloy CRD registration, destination credentials, and the relevant storage class or hostPath mounts.
• Related runbook: ../runbooks/k8s-monitoring.md

8. Release and Change Notes

• Current deployed app version: chart 4.5.0 on all three clusters.
• Current chart version: grafana/k8s-monitoring 4.5.0 , pinned once in fleet.yaml and in each overlays//kustomization.yaml.
• Last significant change: restructured to the dual-mode Helm pattern (chart inflated live by Fleet or kustomize --enable-helm, no committed rendered output), vendored the Alloy CRD pinned to alloy-operator 0.6.3, and upgraded the chart from the stale 4.1.4 committed renders (whose regeneration comments claimed 4.0.1) to 4.5.0.
• Rollback reference: revert the affected overlay values.yaml and kustomization.yaml revision, then rebuild the overlay with Kustomize and Helm enabled.