Skip to content

prometheus

Metadata

Field Value
Service prometheus
Purpose Metrics collection, alerting, and time-series storage
Criticality Tier 1
Owner Platform / Observability owner
Clusters jls
Namespace prometheus
Exposure internet
Stateful yes
Backup class snapshot
RPO / RTO Daily backup target, 2 to 6 hours to restore
Last reviewed 2026-08-24

1. Service Overview

Prometheus provides metrics collection, retention, and alert delivery for the JLS cluster deployment represented in this repository.

Summary

If Prometheus fails, metrics queries and alerting stop for the affected environment.

Dependencies

Dependency Type Why it matters
Alertmanager alerting Receives and routes alerts
Traefik ingress Exposes Prometheus and Alertmanager where configured
Persistent storage storage Stores TSDB data and alerting state

2. Architecture Diagram

[Alloy / k8s-monitoring] --remote-write--> [Prometheus] --remote-write--> [Mimir monolith] --> S3
[static targets: mimir-monolith, self]         |
                                               v
                                  [prometheus-alertmanager] --> forgejo ticket bridge
  • Prometheus is the rule/alert evaluation point and short-term store (15d). Mimir is the long-term store; Grafana queries Mimir.
  • Alert delivery goes to exactly one Alertmanager (the chart's own prometheus-alertmanager); Mimir's in-process alertmanager is not running.
  • A filtered low-cardinality subset is remote-written to Grafana Cloud as a backup view.

3. Deployment Specifications

Item Value
Source path prometheus/overlays/jls
Deployment model Dual-mode Helm: Fleet inflates the chart natively then applies overlays/jls/fleet as post-render; local rendering uses kustomize --enable-helm on the same values file
Namespace prometheus
Workload kind StatefulSets and Deployments
Chart or image version prometheus chart 29.10.0, pinned in fleet.yaml and overlays/jls/kustomization.yaml (keep in sync)
Config files overlays/jls/values.yaml (single source of truth) plus the shared overlays/jls/fleet/ layer

Cluster mapping

Cluster Overlay path Notes
jls prometheus/overlays/jls Current JLS deployment

4. Configuration Guide

Environment variables

Variable Source Purpose Secret?
Helm values-driven settings rendered chart values and secrets Configure scraping, alerting, and external integrations mixed

ConfigMaps

Resource Path Purpose
Helm-generated ConfigMaps prometheus/overlays/jls Rule files, scrape config, and chart runtime config

Secrets management

  • Secret names: alerting, remote-write, and ingress-related secrets in the prometheus namespace
  • Source of truth: chart values and runtime secret material
  • Rotation trigger: remote-write or alert receiver changes
  • Recovery note: restore alerting and remote-write secrets before restarting pods

5. Access Protocols

Path URL or endpoint Audience Auth TLS terminates at
Internal Prometheus and Alertmanager services in the namespace Platform workloads cluster RBAC Service / ingress
External https://prometheus.mutana.site and related ingress endpoints Operators ingress auth policy Traefik

6. Operations and Observability

  • Primary health indicators: scrape targets healthy, TSDB writable, and alerts delivered.
  • Dashboards or alerts: shared Grafana plus Prometheus self-monitoring.
  • Log locations: Prometheus and Alertmanager pod logs.
  • Known failure modes: disk pressure, scrape misconfiguration, alert receiver errors, or ingress failures.

7. Backup and Recovery Notes

  • Backup method: TSDB PVC snapshot and alerting config backup.
  • Restore prerequisites: restored persistent storage and runtime secrets.
  • Related runbook: ../runbooks/prometheus.md

8. Release and Change Notes

  • Current deployed app version: see the chart pin (29.10.0) in fleet.yaml.
  • Current chart version: prometheus-community/prometheus 29.10.0.
  • Last significant change: restructured to the dual-mode Helm pattern (defectdojo shape); no rendered chart output is committed anymore.
  • Rollback reference: previous overlay revision in Git.