Skip to content

grafana-dashboard

Metadata

Field Value
Service grafana-dashboard
Purpose Grafana for the local cluster with provisioned datasources and Git Sync dashboards
Criticality Tier 2
Owner Platform / Observability owner
Clusters local
Namespace grafana
Exposure internet
Stateful yes
Backup class snapshot
RPO / RTO Git is authoritative for dashboards and datasources; PVC snapshot for users/preferences, minutes to restore
Last reviewed 2026-08-26

1. Service Overview

This service runs Grafana, provisions its datasources at bootstrap, and keeps dashboards synchronized with the dashboards/ folder of this repository through Grafana Git Sync.

Summary

If it fails, cluster dashboards and metric/log exploration are unavailable; the monitoring stack (Prometheus, Loki on the JLS cluster) keeps collecting.

Dependencies

Dependency Type Why it matters
Prometheus (JLS cluster) external datasource https://prometheus.mutana.site, basic auth promuser
Loki (JLS cluster) external datasource https://loki.mutana.site, basic auth lokiuser, tenant header injected by Traefik
GitHub (arnaudmut/k8s) provisioning Git Sync pulls grafana-dashboard/dashboards/ from main with a PAT
Traefik ingress Exposes the Grafana UI at https://grafana.mutana.fr

2. Architecture Diagram

[arnaudmut/k8s dashboards/] --Git Sync (PAT)--> [Grafana (ns grafana)] --> [Traefik] --> users
                                                      |
                                                      +--> https://prometheus.mutana.site (Prometheus, JLS)
                                                      +--> https://loki.mutana.site (Loki, JLS)

3. Deployment Specifications

Item Value
Source path grafana-dashboard/base and grafana-dashboard/overlays/local
Deployment model Kustomize plus Fleet bundle
Namespace grafana
Workload kind Deployment (grafana), Job (grafana-git-sync-setup)
Image grafana/grafana:13.0.1-security-01, pinned in the overlay images: block
Config files base/kustomization.yaml, overlays/local/kustomization.yaml, fleet.yaml

Cluster mapping

Cluster Overlay path Notes
local grafana-dashboard/overlays/local Only target

4. Configuration Guide

Datasources (bootstrap provisioning)

The grafana-datasources ConfigMap is mounted at /etc/grafana/provisioning/datasources. Passwords are expanded from PROMETHEUS_PASSWORD / LOKI_PASSWORD env vars sourced from the grafana-secret Secret keys prom-password / loki-password. Datasource UIDs (prometheus-oci, loki-oci) are referenced by dashboards and must not be renamed.

Dashboards (Git Sync)

The grafana-git-sync-setup Job creates the Grafana Repository resource (GitHub, branch main, path grafana-dashboard/dashboards/, sync every 60s, write workflow enabled so UI saves commit back to Git).

Secrets management

  • grafana-secret: admin-user, admin-password, prom-password, loki-password. Generated from the tracked overlays/local/grafana-secrets.env (known tracked-debt file).
  • grafana-github-pat: github-pat for Git Sync. Generated from the tracked overlays/local/grafana-github-pat.env.
  • Rotation: update the env file, rotate the PAT on GitHub, then re-apply; for Git Sync, delete the grafana-git-sync-setup Job to force re-setup.

5. Access Protocols

Path URL or endpoint Audience Auth TLS terminates at
External https://grafana.mutana.fr Operators and users Grafana admin/login Traefik

6. Operations and Observability

  • Primary health indicators: grafana pod ready, datasources healthy in the UI, Git Sync repository status green.
  • Log locations: kubectl -n grafana logs deploy/grafana.
  • Known failure modes: expired GitHub PAT, changed datasource credentials on the JLS ingress, invalid dashboard JSON pushed to main.

7. Backup and Recovery Notes

  • Backup method: Git is the source of truth for dashboards and datasources; the grafana-pvc snapshot covers users and preferences.
  • Restore prerequisites: grafana-secret and grafana-github-pat values, reachable Prometheus/Loki endpoints.
  • Related runbook: ../runbooks/grafana-dashboard.md

8. Release and Change Notes

  • Current deployed app version: grafana/grafana:13.0.1-security-01.
  • Last significant change: 2026-08-26 streamlining — datasources repointed to the mutana.site endpoints with env-var passwords (removed dead Alertmanager entry and mutana.fr URLs), dead bootstrap job and duplicated dashboard copies removed, image pinned.
  • Rollback reference: previous dashboard overlay revision in Git.