mimir Runbook¶
Metadata¶
| Field | Value |
|---|---|
| Cluster / namespace | cc-obs-kube / mimir |
| Workload | StatefulSet mimir-monolith |
| Service page | Mimir |
| Last reviewed | 2026-10-20 |
Trigger Conditions¶
The Mimir Fleet bundle is applied but unready, remote-write fails, or queries fail. Separate pod readiness, storage access and public ingress.
1. Health Checks¶
kubectl --context cc-obs-kube -n mimir get sts,pods,svc,pvc -o wide
kubectl --context cc-obs-kube -n mimir describe pod mimir-monolith-0
kubectl --context cc-obs-kube -n mimir get secret mimir-garage-credentials
# Once the process is running, in separate terminals:
kubectl --context cc-obs-kube -n mimir port-forward svc/mimir-monolith 18080:8080
curl --fail http://127.0.0.1:18080/ready
Secret metadata is sufficient to check existence; do not print its data.
2. Troubleshooting Workflows¶
Inspect current or previous pod logs locally and redact credentials before sharing them. Access Denied during the startup object-storage check requires approved bucket permissions and matching endpoint/region configuration. Repeated pod restarts or Fleet resyncs do not repair S3 authorization.
On 2026-10-20 the owner approved Garage with fresh buckets. Check
garage.garage.svc.cluster.local:3900, region garage, HTTP/path-style and
mimir-garage-credentials. Run the Garage health and S3 checks
before restarting Mimir. The old external configuration previously denied access;
reverting to it requires restored permissions and a separate data plan.
The initial ingester readiness stabilization can take about 90 seconds. Existing
Prometheus remote-write queues resume on their retry schedule; a successful
/ready alone does not prove that queued samples have arrived. Verify ingest
counters and a current PromQL query. Backlog replay can hit the configured
rate limit; inspect queue lag and retries before changing ingestion policy.
A diagnostic /ingester/flush?tenant=anonymous&wait=true can take longer than
the HTTP client's timeout while background shipping continues. Inspect shipper
counters and Garage object counts before declaring the flush failed. It can
briefly reject writes during compaction, so avoid repeated diagnostic flushes.
3. Disaster Recovery¶
Preserve data-mimir-monolith-0 and the object-store contents. Restore the
approved configuration and credential references, then recover local state
and blocks using the owner's backup procedure. Validate readiness, ingestion
and a query for existing metrics. This incident did not test a data restore.
4. Scaling and Resource Management¶
Keep the existing single-replica monolith and replication configuration. Review resource pressure before changing requests or limits. A distributed deployment or additional replicas requires a separate storage and topology plan.
5. Maintenance Procedures¶
Build mimir/overlays/jls with Kustomize, discarding rendered Secret output.
Use the normal credential-management workflow for rotation. Confirm the
target cluster and preserve PVC, Service and Secret names during upgrades.
6. Rollback Strategy¶
Revert the affected configuration commit through the normal PR workflow and verify StatefulSet readiness. Retain the PVC. An S3 endpoint or bucket rollback also requires a data-consistency plan; reverting YAML alone does not move blocks.
7. Post-Incident Actions¶
Record the approved object-store configuration location and verification results in the Fleet incident runbook. Keep credentials out of Git and incident notes, and update the changelog for deployment changes.