Skip to content

prometheus

Metadata

Field Value
Service prometheus
Purpose Metrics collection, alerting, and time-series storage
Criticality Tier 1
Owner Platform / Observability owner
Clusters jls
Namespace prometheus
Exposure internet
Stateful yes
Backup class snapshot
RPO / RTO Daily backup target, 2 to 6 hours to restore
Last reviewed 2026-08-24

1. Service Overview

Prometheus provides metrics collection, retention, and alert delivery for the JLS cluster deployment represented in this repository.

Summary

If Prometheus fails, metrics queries and alerting stop for the affected environment.

Dependencies

Dependency Type Why it matters
Alertmanager alerting Receives and routes alerts
Traefik ingress Exposes Prometheus and Alertmanager where configured
Persistent storage storage Stores TSDB data and alerting state

2. Architecture Diagram

[Alloy / k8s-monitoring] --remote-write--> [Prometheus] --remote-write--> [Mimir monolith] --> S3
[static targets: mimir-monolith, self]         |
                                               v
                                  [prometheus-alertmanager] --> forgejo ticket bridge
  • Prometheus is the rule/alert evaluation point and short-term store (15d). Mimir is the long-term store; Grafana queries Mimir.
  • Alert delivery goes to exactly one Alertmanager (the chart's own prometheus-alertmanager); Mimir's in-process alertmanager is not running.
  • A filtered low-cardinality subset is remote-written to Grafana Cloud as a backup view.

3. Deployment Specifications

Item Value
Source path prometheus/overlays/jls
Deployment model Dual-mode Helm: Fleet inflates the chart natively then applies overlays/jls/fleet as post-render; local rendering uses kustomize --enable-helm on the same values file
Namespace prometheus
Workload kind StatefulSets and Deployments
Chart or image version prometheus chart 29.25.0 , pinned in fleet.yaml and overlays/jls/kustomization.yaml (keep in sync)
Config files overlays/jls/values.yaml (single source of truth) plus the shared overlays/jls/fleet/ layer

Cluster mapping

Cluster Overlay path Notes
jls prometheus/overlays/jls Current JLS deployment

4. Configuration Guide

Environment variables

Variable Source Purpose Secret?
Helm values-driven settings rendered chart values and secrets Configure scraping, alerting, and external integrations mixed

ConfigMaps

Resource Path Purpose
Helm-generated ConfigMaps prometheus/overlays/jls Rule files, scrape config, and chart runtime config

Recording rules

serverFiles.recording_rules.yml carries four groups:

Group Purpose Forwarded to Grafana Cloud
infrastructure.recording Low-cardinality node aggregates yes
kubernetes.recording Low-cardinality cluster aggregates yes
kubernetes-mixin.recording Series the vendored Kubernetes dashboards query by name no
prometheus.recording Remote-write pipeline health yes

kubernetes-mixin.recording exists because the dashboards in grafana-dashboard/dashboards/Kubernetes are the kube-prometheus mixin "Compute Resources" boards: they query node_namespace_pod_container:*, namespace_cpu:*, namespace_memory:* and namespace_workload_pod:kube_pod_owner:relabel directly. Nothing else in this stack produces them — the k8s-monitoring chart only forwards raw metrics, and kube-prometheus-stack is not deployed on any cluster.

These rules are per namespace and per pod. The grafana-cloud remote-write write_relabel_configs is an explicit keep list budgeted for under 5 000 active series, so leaving them out of it is deliberate: adding them there would break that budget.

They also depend on kube-state-metrics series actually arriving. If they evaluate empty, check the Alloy keep-rule before the rules themselves — see k8s-monitoring.md §4, kube-state-metrics tuning.

Secrets management

  • Secret names: alerting, remote-write, and ingress-related secrets in the prometheus namespace
  • Source of truth: chart values and runtime secret material
  • Rotation trigger: remote-write or alert receiver changes
  • Recovery note: restore alerting and remote-write secrets before restarting pods

5. Access Protocols

Path URL or endpoint Audience Auth TLS terminates at
Internal Prometheus and Alertmanager services in the namespace Platform workloads cluster RBAC Service / ingress
External https://prometheus.mutana.site and related ingress endpoints Operators ingress auth policy Traefik

6. Operations and Observability

  • Primary health indicators: scrape targets healthy, TSDB writable, and alerts delivered.
  • Dashboards or alerts: shared Grafana plus Prometheus self-monitoring.
  • Log locations: Prometheus and Alertmanager pod logs.
  • Known failure modes: disk pressure, scrape misconfiguration, alert receiver errors, or ingress failures.

7. Backup and Recovery Notes

  • Backup method: TSDB PVC snapshot and alerting config backup.
  • Restore prerequisites: restored persistent storage and runtime secrets.
  • Related runbook: ../runbooks/prometheus.md

8. Release and Change Notes

  • Current deployed app version: see the chart pin 29.25.0 in fleet.yaml.
  • Current chart version: prometheus-community/prometheus 29.25.0 .
  • Last significant change: restructured to the dual-mode Helm pattern (defectdojo shape); no rendered chart output is committed anymore.
  • Rollback reference: previous overlay revision in Git.