Skip to content

grafana-dashboard Runbook

Metadata

Field Value
Service grafana-dashboard
Criticality Tier 2
Owner Platform / Observability owner
Namespace grafana
Clusters local
Last validated 2026-09-17
Related service page ../services/grafana-dashboard.md

Trigger Conditions

  • Expected dashboards are missing or stale.
  • Git Sync fails to update dashboard content.
  • Grafana provisioning logs show parse or auth errors.
  • Datasource queries fail (panels empty).

1. Health Checks

kubectl -n grafana get pods,configmap,secret
kubectl -n grafana logs deploy/grafana --tail=200
curl -s https://grafana.mutana.fr/api/health

2. Troubleshooting Workflows

Datasources

Datasources are provisioned from the grafana-datasources ConfigMap; passwords come from grafana-secret (prom-password, loki-password) via the PROMETHEUS_PASSWORD / LOKI_PASSWORD env vars.

kubectl -n grafana logs deploy/grafana --tail=400 | grep -i provision
# verify credentials against the upstream endpoints directly:
curl -s https://prometheus.mutana.site/api/v1/query -u "promuser:<prom-password>" --data-urlencode 'query=up'
curl -s "https://loki.mutana.site/loki/api/v1/labels" -u "lokiuser:<loki-password>"

Restart the Deployment after fixing the Secret or ConfigMap so provisioning re-runs. Note: the Loki tenant header (X-Scope-OrgID: homelab) is injected by the loki-header Traefik middleware on the JLS ingress — it is not set in the datasource.

Git Sync dashboards

Dashboards come from grafana-dashboard/dashboards/ on main of github.com/arnaudmut/k8s. The grafana-git-sync-setup Job (idempotent, ttl-cleaned) creates the Grafana Repository resource.

Git Sync reads GitHub, never the local worktree. Editing a dashboard file without committing and pushing changes nothing in Grafana; the sync also skips entirely while the commit SHA is unchanged (skip incremental sync as reference is the same in the controller log), so a push is what actually applies a change.

The Repository status is the authoritative list of what failed — read it before guessing:

G=https://grafana.mutana.fr
U=$(kubectl -n grafana get secret grafana-secret -o jsonpath='{.data.admin-user}' | base64 -d)
P=$(kubectl -n grafana get secret grafana-secret -o jsonpath='{.data.admin-password}' | base64 -d)

# sync state, per-file messages, and resource counts
curl -sk -u "$U:$P" \
  "$G/apis/provisioning.grafana.app/v0alpha1/namespaces/default/repositories" \
  | python3 -c 'import sys,json; s=json.load(sys.stdin)["items"][0]["status"]; print(s["sync"]["state"]); [print(" -",m) for m in s["sync"].get("message",[])]; print(s["stats"])'

Force a full re-pull without touching Git by bumping the sync interval (the generation change makes the controller re-read the whole tree):

curl -sk -u "$U:$P" -X PATCH -H 'Content-Type: application/merge-patch+json' \
  -d '{"spec":{"sync":{"intervalSeconds":61}}}' \
  "$G/apis/provisioning.grafana.app/v0alpha1/namespaces/default/repositories/repository-f043629"
# then patch it back to 60

For PAT rotation or a broken Repository, re-run the setup Job:

kubectl -n grafana delete job grafana-git-sync-setup
kubectl -n grafana logs job/grafana-git-sync-setup

The PAT (grafana-github-pat Secret) needs scopes Contents read/write, Metadata read, Pull requests read/write.

Failure modes seen in this repository

Symptom in the sync message Cause Fix
folder "X/" is missing folder metadata file the directory has no _folder.json; Grafana invents an unstable UID add _folder.json (see the README)
resource '<uid>' already exists and is not managed; cannot take over without an explicit migration a dashboard with that UID was created by hand in Grafana before the file was added to Git back up the dashboard, delete the unmanaged copy, then force a full re-pull so Git recreates it in the right folder
dashboards referenced by the repo never appear a file was skipped without an error message — seen with _-and-space, parenthesised, and timestamped export filenames rename to a plain name and push
two files, same metadata.name one file is a redundant export of a dashboard already in the tree; the last writer wins and the other is dropped silently delete the duplicate file

Fleet bundle fails with illegal name ... in file creation

Fleet bundles the whole grafana-dashboard tree (not just the kustomized base/ + overlays/local), and its post-render refuses to create files or directories whose names contain spaces. A UI export saved with its default filename (SNMP fleet overview-1789677257171.json) therefore blocks the entire bundle, grafana-git-sync-setup included. dashboards/ is listed in grafana-dashboard/.fleetignore, so new exports cannot trigger this — it only reappears if someone re-includes the folder. Rename offenders (space -> hyphen) and push.

grafana-git-sync-setup crashes with HTTP Error 403 (Cloudflare error code: 1010)

grafana.ini sets enforce_domain = true: an API request whose Host is not grafana.mutana.fr gets a 301 to root_url. urllib followed that redirect out of the cluster, where Cloudflare blocks the python user-agent. The job now sends Host: grafana.mutana.fr while dialing http://grafana in-cluster (GRAFANA_HOST env). Keep that header if the job's script is ever rewritten — any in-cluster Grafana API client here needs it. The same applies to hand-rolled probes: plain curl http://grafana/api/... from a pod fails with 403/1010 for the same reason; pass -H 'Host: grafana.mutana.fr' (or use the public URL, which works for curl because Cloudflare does not block its user-agent).

Panels show "no data" although metrics exist

Datasources here are provisioned (prometheus-oci, loki-oci) but dashboards imported from Grafana Cloud carry their own datasource variable — usually ${datasource} or ${prometheus_datasource} / ${loki_datasource}. If that variable's current.value is empty, every panel resolves no datasource and renders empty.

# which datasource does a live dashboard actually resolve?
curl -sk -u "$U:$P" "$G/api/dashboards/uid/<uid>" \
  | python3 -c 'import sys,json; d=json.load(sys.stdin)["dashboard"]; [print(v.get("name"), (v.get("current") or {}).get("value")) for v in d.get("templating",{}).get("list",[]) if v.get("type")=="datasource"]'

Fix the value in the dashboard file, not in the UI: a UI edit is written back by the write workflow only for resources Git already manages.

Homelab dashboard set

The dashboards/Homelab/ set is generated. Before pushing a change to it, run both checkers from grafana-dashboard/:

# offline: JSON, duplicate UIDs, filenames, datasources, PromQL syntax
python3 tools/validate_dashboards.py

# live: every expression, against Prometheus and Loki
export PROM_PW=$(grep '^prom-password=' overlays/local/grafana-secrets.env | cut -d= -f2-)
export LOKI_PW=$(grep '^loki-password=' overlays/local/grafana-secrets.env | cut -d= -f2-)
python3 tools/validate_queries.py

validate_queries.py is the one that catches a panel which parses but matches nothing — the shape a dashboard takes when a metric is renamed or a keep-list drops it. It retries through prometheus-server restarts, so a 404 from Traefik does not produce false failures.

A panel reported as returning no data is not always a bug: the "what's broken" tables (Containers not running, Deployments below desired replicas, DaemonSets not fully scheduled, Entities unavailable) are empty when the estate is healthy and carry a noValue message saying so.

To change a dashboard, edit tools/gen_part1.py or tools/gen_part2.py and re-run python3 tools/gen_dashboards.py. This overwrites the JSON and discards anything saved from the Grafana UI since the last run. If a dashboard has been edited in the UI, either port the change back into the generator or stop generating that file — not both.

prometheus.mutana.site returns 404 page not found

Traefik answers 404, not 503, when an IngressRoute's service has no ready endpoints, so a 404 on every path (including /) means the backend is down rather than the route being wrong. Confirm with:

kubectl --context jls -n prometheus get pods
kubectl --context jls -n prometheus get endpoints prometheus-server

An empty ENDPOINTS column with a 1/2 Running pod is prometheus-server failing its readiness probe. Seen on 2026-09-18 with 367 restarts, caused by the liveness probe (/-/healthy, timeoutSeconds: 10) timing out while the TSDB compacted — compaction was taking 4m43s against overlapping blocks while remote-write to Mimir was backing up (failed pushing to ingester mimir-monolith-0: context deadline exceeded). Every Grafana dashboard is blank while this lasts.

3. Disaster Recovery

  1. Re-apply the overlay (kubectl apply -k grafana-dashboard/overlays/local); datasources and the Git Sync setup Job are recreated automatically.
  2. Restore grafana-secret / grafana-github-pat values if lost (Git holds the generator env files).
  3. Dashboards re-sync from Git; users and preferences come from the grafana-pvc snapshot if needed.
  4. Verify dashboard folders and datasource health in the UI.

4. Scaling and Resource Management

kubectl -n grafana top pod

Grafana here needs config fixes rather than scaling; raise resources only if the pod is OOM-killed.

5. Maintenance Procedures

  • Rotate the GitHub PAT: update grafana-github-pat.env, re-apply, delete the grafana-git-sync-setup Job to force re-setup.
  • Rotate datasource passwords: update grafana-secrets.env, re-apply (the Deployment env change rolls the pod).
  • Validate dashboard JSON before pushing to main; broken JSON makes the sync branch fail silently in the UI. Before pushing, check that every folder has _folder.json, no two files share a metadata.name/uid, and every datasource variable resolves to prometheus-oci or loki-oci.
  • After a push, read the Repository status (above) rather than the UI; it names the file that failed and why.

6. Rollback Strategy

  • git revert the change set and re-apply the overlay.
  • Restart Grafana if provisioning caches stale content.

7. Post-Incident Actions

  1. Add a changelog fragment if manual remediation was required.
  2. Update the service page if provisioning behavior changed.
  3. Extend this runbook when a new failure mode appears.

Homelab landing page maintenance

homelab-landing is API-managed, unlike the Git-synced imported dashboards. Rebuild with python3 grafana-dashboard/landing/build.py, review home.json, and publish using python3 grafana-dashboard/landing/publish.py --set-home. The publisher captures the previous dashboard and organization preferences for rollback before changes. See grafana-dashboard/landing/README.md for the rollback fields and the ownership migration caveat. An individual's custom home dashboard takes precedence over the organization home.

If links work but all metrics fail, check datasource health and the upstream Prometheus route before changing the dashboard. Missing telemetry is not mapped to a healthy value; an upstream 404 affects the existing dashboards too.

For the Linux / DevOps edition, validate instant queries for snapshot panels and six-hour range queries for trend panels through the prometheus-oci datasource proxy. A syntactically successful query without finite samples is not a populated chart. Check node_pressure_* and node_zfs_arc_* exporter coverage if those panels alone are empty. Host availability tracks scrapes, not service health; do not interpret blank timeline segments as uptime. Application shortcuts remain available below the graphs even during a metrics outage.