loki
| Field |
Value |
| Service |
loki |
| Purpose |
Shared log aggregation and query backend |
| Criticality |
Tier 1 |
| Owner |
Platform / Observability owner |
| Clusters |
jls |
| Namespace |
loki |
| Exposure |
internet |
| Stateful |
yes |
| Backup class |
app-native |
| RPO / RTO |
Daily storage target, 2 to 6 hours to restore depending on backend mode |
| Last reviewed |
2026-05-20 |
1. Service Overview
Loki stores and serves application and platform logs for the monitored clusters.
Summary
If Loki fails, log ingestion and historical log search disappear for the affected environments.
Dependencies
| Dependency |
Type |
Why it matters |
| Object storage or persistent volumes |
storage |
Stores chunks, indexes, and compactor state |
| Traefik |
ingress |
Exposes Loki APIs where configured |
| Collectors and monitoring stack |
ingest path |
Feed logs into the backend |
2. Architecture Diagram
[Collectors]
-> [Loki gateway / distributors]
-> [Loki storage components]
-> [Grafana / log queries]
3. Deployment Specifications
| Item |
Value |
| Source path |
loki/overlays/jls |
| Deployment model |
Dual-mode Helm: Fleet inflates the chart natively then applies overlays/jls/fleet as post-render; local rendering uses kustomize --enable-helm on the same values file |
| Namespace |
loki |
| Workload kind |
StatefulSet (monolithic/single-binary mode) |
| Chart or image version |
loki chart 18.11.3 / Loki 3.7.6 (OCI: ghcr.io/grafana-community/helm-charts), pinned in fleet.yaml and overlays/jls/kustomization.yaml (keep in sync) |
| Config files |
overlays/jls/values.yaml (single source of truth) plus the shared overlays/jls/fleet/ layer |
Cluster mapping
| Cluster |
Overlay path |
Notes |
| jls |
loki/overlays/jls |
Only active deployment; monolithic mode |
4. Configuration Guide
Environment variables
| Variable |
Source |
Purpose |
Secret? |
| Loki runtime settings |
Helm values and overlay manifests |
Configure gateways, storage, retention, and scaling |
mixed |
ConfigMaps
| Resource |
Path |
Purpose |
| Helm-generated runtime config |
loki/* values and manifests |
Configure the deployed Loki mode |
Secrets management
- Secret names: storage credentials and optional auth secrets in the loki namespace
- Source of truth: values files plus runtime secrets
- Rotation trigger: storage credential changes or endpoint rotation
- Recovery note: restore object-store credentials before restarting ingestion components
5. Access Protocols
| Path |
URL or endpoint |
Audience |
Auth |
TLS terminates at |
| Internal |
Loki services in the loki namespace |
Collectors and Grafana |
cluster RBAC / service auth |
Service / gateway |
| External |
https://loki.mutana.site where enabled |
Operators and integrations |
ingress auth policy |
Traefik |
6. Operations and Observability
- Primary health indicators: ingestion succeeds, queries return, and storage backends are healthy.
- Dashboards or alerts: shared Grafana and Loki health dashboards.
- Log locations: gateway and backend pod logs in the loki namespace.
- Known failure modes: object-store auth failures, compactor drift, retention misconfiguration, or ingress breakage.
7. Backup and Recovery Notes
- Backup method: backend storage snapshots or object-store policy.
- Restore prerequisites: restored storage credentials and healthy compactor/query paths.
- Related runbook: ../runbooks/loki.md
8. Release and Change Notes
- Current deployed app version: see each environment's committed chart version.
- Current chart version: environment-specific under the loki directory.
- Last significant change: documentation coverage added for the multi-environment Loki layout and Fleet entry points.
- Rollback reference: previous Helm values or environment revision in Git.