Skip to content

grafana-dashboard Runbook

Metadata

Field Value
Service grafana-dashboard
Criticality Tier 2
Owner Platform / Observability owner
Namespace grafana
Clusters local
Last validated 2026-08-26
Related service page ../services/grafana-dashboard.md

Trigger Conditions

  • Expected dashboards are missing or stale.
  • Git Sync fails to update dashboard content.
  • Grafana provisioning logs show parse or auth errors.
  • Datasource queries fail (panels empty).

1. Health Checks

kubectl -n grafana get pods,configmap,secret
kubectl -n grafana logs deploy/grafana --tail=200
curl -s https://grafana.mutana.fr/api/health

2. Troubleshooting Workflows

Datasources

Datasources are provisioned from the grafana-datasources ConfigMap; passwords come from grafana-secret (prom-password, loki-password) via the PROMETHEUS_PASSWORD / LOKI_PASSWORD env vars.

kubectl -n grafana logs deploy/grafana --tail=400 | grep -i provision
# verify credentials against the upstream endpoints directly:
curl -s https://prometheus.mutana.site/api/v1/query -u "promuser:<prom-password>" --data-urlencode 'query=up'
curl -s "https://loki.mutana.site/loki/api/v1/labels" -u "lokiuser:<loki-password>"

Restart the Deployment after fixing the Secret or ConfigMap so provisioning re-runs. Note: the Loki tenant header (X-Scope-OrgID: homelab) is injected by the loki-header Traefik middleware on the JLS ingress — it is not set in the datasource.

Git Sync dashboards

Dashboards come from grafana-dashboard/dashboards/ on main of github.com/arnaudmut/k8s. The grafana-git-sync-setup Job (idempotent, ttl-cleaned) creates the Grafana Repository resource.

# force re-setup (e.g. after PAT rotation or Repository misconfiguration):
kubectl -n grafana delete job grafana-git-sync-setup
kubectl -n grafana logs job/grafana-git-sync-setup

Check the Git Sync status in the Grafana UI (Connections / Git sync) and the grafana-github-pat Secret (scopes: Contents read/write, Metadata read, Pull requests read/write).

3. Disaster Recovery

  1. Re-apply the overlay (kubectl apply -k grafana-dashboard/overlays/local); datasources and the Git Sync setup Job are recreated automatically.
  2. Restore grafana-secret / grafana-github-pat values if lost (Git holds the generator env files).
  3. Dashboards re-sync from Git; users and preferences come from the grafana-pvc snapshot if needed.
  4. Verify dashboard folders and datasource health in the UI.

4. Scaling and Resource Management

kubectl -n grafana top pod

Grafana here needs config fixes rather than scaling; raise resources only if the pod is OOM-killed.

5. Maintenance Procedures

  • Rotate the GitHub PAT: update grafana-github-pat.env, re-apply, delete the grafana-git-sync-setup Job to force re-setup.
  • Rotate datasource passwords: update grafana-secrets.env, re-apply (the Deployment env change rolls the pod).
  • Validate dashboard JSON before pushing to main; broken JSON makes the sync branch fail silently in the UI.

6. Rollback Strategy

  • git revert the change set and re-apply the overlay.
  • Restart Grafana if provisioning caches stale content.

7. Post-Incident Actions

  1. Add a changelog fragment if manual remediation was required.
  2. Update the service page if provisioning behavior changed.
  3. Extend this runbook when a new failure mode appears.