Skip to content

grafana-dashboard

Metadata

Field Value
Service grafana-dashboard
Purpose Grafana for the local cluster with provisioned datasources and Git Sync dashboards
Criticality Tier 2
Owner Platform / Observability owner
Clusters local
Namespace grafana
Exposure internet
Stateful yes
Backup class snapshot
RPO / RTO Git is authoritative for dashboards and datasources; PVC snapshot for users/preferences, minutes to restore
Last reviewed 2026-09-17

1. Service Overview

This service runs Grafana, provisions its datasources at bootstrap, and keeps dashboards synchronized with the dashboards/ folder of this repository through Grafana Git Sync.

Summary

If it fails, cluster dashboards and metric/log exploration are unavailable; the monitoring stack (Prometheus, Loki on the JLS cluster) keeps collecting.

Dependencies

Dependency Type Why it matters
Prometheus (JLS cluster) external datasource https://prometheus.mutana.site, basic auth promuser
Loki (JLS cluster) external datasource https://loki.mutana.site, basic auth lokiuser, tenant header injected by Traefik
GitHub (arnaudmut/k8s) provisioning Git Sync pulls grafana-dashboard/dashboards/ from main with a PAT
Traefik ingress Exposes the Grafana UI at https://grafana.mutana.fr

2. Architecture Diagram

[arnaudmut/k8s dashboards/] --Git Sync (PAT)--> [Grafana (ns grafana)] --> [Traefik] --> users
                                                      |
                                                      +--> https://prometheus.mutana.site (Prometheus, JLS)
                                                      +--> https://loki.mutana.site (Loki, JLS)

3. Deployment Specifications

Item Value
Source path grafana-dashboard/base and grafana-dashboard/overlays/local
Deployment model Kustomize plus Fleet bundle
Namespace grafana
Workload kind Deployment (grafana), Job (grafana-git-sync-setup)
Image grafana/grafana:13.0.1-security-01 , pinned in the overlay images: block
Config files base/kustomization.yaml, overlays/local/kustomization.yaml, fleet.yaml

Cluster mapping

Cluster Overlay path Notes
local grafana-dashboard/overlays/local Only target

4. Configuration Guide

Datasources (bootstrap provisioning)

The grafana-datasources ConfigMap is mounted at /etc/grafana/provisioning/datasources. Passwords are expanded from PROMETHEUS_PASSWORD / LOKI_PASSWORD env vars sourced from the grafana-secret Secret keys prom-password / loki-password. Datasource UIDs (prometheus-oci, loki-oci) are referenced by dashboards and must not be renamed.

Dashboards (Git Sync)

The grafana-git-sync-setup Job creates the Grafana Repository resource (GitHub, branch main, path grafana-dashboard/dashboards/, sync every 60s, write workflow enabled so UI saves commit back to Git). The sync reads GitHub, not this repository's worktree, and does nothing while the commit SHA is unchanged — a dashboard edit only takes effect once pushed.

The tree under dashboards/ follows rules Grafana depends on: one directory per folder and every directory carries a _folder.json (the folder UID lives in metadata.name); metadata.name of a dashboard is its UID and must be unique; filenames stay plain. Dashboards imported from Grafana Cloud bind their panels to a datasource variable, so each file must set that variable's current to prometheus-oci or loki-oci — an empty value is the usual cause of a dashboard that loads but shows nothing. The full failure-mode list is in the runbook.

The OPNsense/ folder holds OPNsense / Firewall (uid opnsense-firewall), a 60-panel drill-down on the firewall: the pf state table and its counters, interface throughput with errors and discards, gateway and upstream reachability, the Unbound resolver, and the host underneath. Unlike the dashboards imported from Grafana Cloud it was authored here, so it queries the three collectors that scrape the firewall directly — integrations/opnsense (os-node_exporter), integrations/opnsense/telegraf (os-telegraf) and integrations/snmp. Its one operator-facing knob is the pf_state_limit textbox variable, which supplies the pf state table's ceiling that the device does not publish as a metric; it defaults to this firewall's live value and needs updating if the state table is ever resized. The authoring script and the metric-provenance notes are in tools/opnsense/.

The Homelab/ folder holds a seven-dashboard set authored here rather than imported: Homelab / Overview (uid homelab-overview), Hypervisors (homelab-hypervisors), Storage & ZFS (homelab-storage), Network Edge (homelab-network), Kubernetes Fleet (homelab-k8s-fleet), Observability Backend (homelab-observability) and Home (homelab-home). They bind panels directly to the prometheus-oci and loki-oci UIDs instead of a datasource variable, so the empty-current failure mode cannot occur; every expression was run against the live backends before the set was committed. Together they cover the three Proxmox hosts, the four Kubernetes clusters, the firewall and switch, the ZFS pools, the Alloy/Mimir/Loki pipeline and Home Assistant.

Homelab / Overview is the operational triage screen — what is up, what is broken, where to look next — and is distinct from homelab-landing, which is the organization home page and a directory of service links. Homelab / Network Edge is the two-device view (firewall plus switch, via SNMP); OPNsense / Firewall remains the firewall drill-down.

The generator and the two validators live in grafana-dashboard/tools/. The JSON is authoritative once written, because Git Sync commits UI saves back to it; re-running the generator discards UI edits. tools/ and landing/ are listed in .fleetignore so Fleet does not materialize developer tooling.

Secrets management

  • grafana-secret: admin-user, admin-password, prom-password, loki-password. Generated from the tracked overlays/local/grafana-secrets.env (known tracked-debt file).
  • grafana-github-pat: github-pat for Git Sync. Generated from the tracked overlays/local/grafana-github-pat.env.
  • Rotation: update the env file, rotate the PAT on GitHub, then re-apply; for Git Sync, delete the grafana-git-sync-setup Job to force re-setup.

5. Access Protocols

Path URL or endpoint Audience Auth TLS terminates at
External https://grafana.mutana.fr Operators and users Grafana admin/login Traefik

6. Operations and Observability

  • Primary health indicators: grafana pod ready, datasources healthy in the UI, Git Sync repository status green.
  • Log locations: kubectl -n grafana logs deploy/grafana.
  • Known failure modes: expired GitHub PAT, changed datasource credentials on the JLS ingress, invalid dashboard JSON pushed to main.

7. Backup and Recovery Notes

  • Backup method: Git is the source of truth for dashboards and datasources; the grafana-pvc snapshot covers users and preferences.
  • Restore prerequisites: grafana-secret and grafana-github-pat values, reachable Prometheus/Loki endpoints.
  • Related runbook: ../runbooks/grafana-dashboard.md

8. Release and Change Notes

  • Current deployed app version: grafana/grafana:13.0.1-security-01.
  • Last significant change: 2026-08-26 streamlining — datasources repointed to the mutana.site endpoints with env-var passwords (removed dead Alertmanager entry and mutana.fr URLs), dead bootstrap job and duplicated dashboard copies removed, image pinned.
  • Rollback reference: previous dashboard overlay revision in Git.

Homelab landing page

The organization home is Homelab · Home, UID homelab-landing, available at https://grafana.mutana.fr/d/homelab-landing. It uses native text, stat, time series, and bar gauge panels, with the existing prometheus-oci datasource. Six service collections link to infrastructure, automation, observability, security, knowledge, and home applications. No new plugin or server setting is required. The source and API publisher are in grafana-dashboard/landing/, outside the Git Sync tree to preserve separate resource ownership.

The Linux / DevOps edition places live graphs before the application directory: CPU modes, memory by host, interface traffic, disk throughput, PSI, host scrape history, Linux distributions, Kubernetes workloads, container CPU and restarts, HTTP responses, and ZFS ARC. Chart descriptions state coverage and aggregation limits. All 35 panels remain native Grafana visualizations.