Garage Runbook

Metadata

Field	Value
Cluster / namespace	cc-obs-kube / garage
Source	garage/overlays/jls
Last reviewed	2026-10-24

Trigger Conditions

Garage is unready, Loki/Mimir report S3 errors, a worker is unavailable, or worker disk space is low. With three copies, one unavailable worker leaves read/write quorum; avoid taking a second worker down during recovery.

1. Health Checks

kubectl --context cc-obs-kube -n garage get sts,pods,pvc,pdb -o wide
python3 garage/scripts/bootstrap.py --context cc-obs-kube health
python3 garage/scripts/verify-s3.py --context cc-obs-kube
kubectl --context cc-obs-kube -n garage exec garage-0 -- /garage status

Expect three storage nodes up and 256/256 partitions OK. The S3 verifier creates unique probe objects, reads them through all three peers, then deletes them. It also requires HTTP 403 for cross-application bucket access. It binds localhost port 13900 temporarily and never prints credentials. Inspect logs locally; never print Secret data, process environments or credential-bearing API replies.

For external S3 access after deployment and DNS setup:

kubectl --context cc-obs-kube -n garage get ingressroute garage
kubectl --context cc-obs-kube -n garage get endpointslice -l kubernetes.io/service-name=garage
curl --max-time 15 -i https://s3.mutana.site/

An unsigned request should return an S3 access-denied response, not a login redirect. Then use an authorized S3 client with endpoint https://s3.mutana.site, region garage and path-style addressing to read an allowed bucket. Keep keys in the client's existing credential store.

For the administration UI after provisioning its token and deploying:

kubectl --context cc-obs-kube -n garage get deployment garage-webui
kubectl --context cc-obs-kube -n garage get ingressroute garage-webui
kubectl --context cc-obs-kube -n garage get service garage-webui garage-admin
curl --max-time 15 -I https://garage.mutana.site/

External unauthenticated users should be sent to Authelia. The existing policy bypasses trusted networks and OPTIONS requests. Sign in with 2FA, confirm the cluster dashboard and existing buckets load, and browse an allowed object. The UI grants full administrative access to users admitted by that policy.

2. Troubleshooting Workflows

Check pod events, local PV affinity/bindings, worker readiness and free disk. A Pending replacement cannot use another worker's local PV. All existing NFS claims share a server; moving Garage metadata there defeats disk independence. S3 clients use region garage, path-style lookup and HTTP on the internal endpoint. Check the application's Garage Secret name and key references. For peer problems, inspect public node IDs and garage-peer DNS; preserve the provider's existing pod network configuration. Liveness intentionally avoids cluster quorum checks so a peer failure does not restart healthy servers.

For external failures, check DNS targets the existing Cloudflare-proxied cluster ingress and TLS uses Traefik's default Cloudflare Origin certificate. A Traefik 404 indicates a missing or mismatched route; 502/504 suggests unavailable backends or blocked connections. Compare Traefik pod IPs with the S3-only /32 rules in garage/overlays/jls/network-policy.yaml: Traefik runs on hostNetwork, so namespace selectors alone do not reliably match these connections. See Kubernetes NetworkPolicy and hostNetwork pods. For S3 signature failures, check region, credentials, path-style addressing, client clock and preservation of the Host header/path. Send requests directly to the HTTPS endpoint so redirects do not change signed requests.

For UI startup failures, check that Secret garage-webui-admin exists with key admin-token, without displaying its data. CreateContainerConfigError often means this prerequisite is missing. For API authorization failures, rerun the idempotent webui-credentials phase to validate the token's identity, expiry and full scope. For connection errors, check the garage-admin Service endpoints and the existing Garage policy allowing admin traffic from namespace garage. For uploads, check available ephemeral disk and /tmp mount access. Update both S3 and UI NetworkPolicies if Traefik's worker IPs change.

3. Disaster Recovery

Keep all PVCs/PVs and the original RPC Secret. Do not run initial bootstrap against existing disks with a newly generated RPC secret or erase metadata. For a pod failure, let the StatefulSet recreate it on its bound worker and check that node identity and layout persist. For a worker replacement, retain the two healthy copies, prepare new local data/metadata volumes, and follow upstream cluster layout management and failure recovery for replacing the node. Review and stage the new layout before applying it; wait for resync completion before retiring old state or another worker. Update PVs, public bootstrap IDs and worker names in Git. Do not point an old PV at an empty replacement disk and assume it contains the former node's data.

For total cluster loss, recover object data, metadata and Secrets from an independent backup. No such backup destination was configured by this rollout. Local metadata snapshots alone cannot restore lost object blocks.

4. Scaling and Resource Management

The approved layout assigns 53,687,091,200 bytes per worker with three copies. All three data PVs therefore provide about 50 GiB usable, not 150 GiB usable. PV sizes and layout capacities do not enforce quotas on the shared root disk. Check filesystem free space and metadata/snapshot growth before expansion. Retention remains the existing Loki/Mimir policy; review measured growth before it exceeds this allocation. Extra replicas or capacity require a revised layout and matching real storage, rather than editing a PVC size alone.

5. Maintenance Procedures

Initial installation only, from the repository root:

kubectl --context cc-obs-kube apply -f garage/base/namespace.yaml
python3 garage/scripts/bootstrap.py --context cc-obs-kube prepare
for worker in 70058 70061 70062; do
  kubectl --context cc-obs-kube -n garage wait --for=jsonpath='{.status.phase}'=Succeeded pod/garage-prepare-$worker --timeout=60s
done
kubectl --context cc-obs-kube apply -k garage/overlays/jls
# All three pods must be Running; readiness waits for the initial layout.
python3 garage/scripts/bootstrap.py --context cc-obs-kube layout
kubectl --context cc-obs-kube apply -k garage/overlays/jls
kubectl --context cc-obs-kube -n garage rollout status sts/garage --timeout=180s
python3 garage/scripts/bootstrap.py --context cc-obs-kube credentials
python3 garage/scripts/bootstrap.py --context cc-obs-kube webui-credentials
python3 garage/scripts/verify-s3.py --context cc-obs-kube
kubectl --context cc-obs-kube -n garage delete pod garage-prepare-70058 garage-prepare-70061 garage-prepare-70062

Commit the recorded public peer IDs before enabling the Fleet bundle. Runtime Secrets remain out of Git. Add garage to the management GitRepo paths once bootstrap succeeds (the normal source is the GitHub main branch):

kubectl --context local -n fleet-default patch gitrepo cc-obs-kube --type=merge -p '{"spec":{"paths":["traefik","authelia","prometheus","garage","mimir","loki","k8s-monitoring"]}}'

The layout phase only applies version 1 to an empty layout; it refuses changed peer identities. The credential phase reuses existing keys/Secrets and does not rotate them. New installations on different workers need reviewed storage manifests and preparation targets. Recreating the entire cluster produces new public node IDs; review the recorded peers before use.

For upgrades, pin and review the Garage release, apply through Fleet, and wait for each pod to be ready before the next disruption. Review six-hour metadata snapshots and disk growth; archive them outside the workers using an approved backup destination. Rotate runtime Secrets through the operator's credential workflow and roll consumers; never put their values in Git or incident notes.

For an existing cluster, run webui-credentials before the reviewed Fleet change adds the UI, then verify deployment/garage-webui becomes Available. DNS for garage.mutana.site must use the cluster's existing Cloudflare-proxied ingress. The bootstrap creates a dedicated dynamic admin API token, sends it directly to Kubernetes Secret creation, and prints no credential values. If the token exists but the Secret was lost (including a failed Secret write), restore it from the operator's backup or revoke the stale token before rerunning bootstrap. Tokens cannot be retrieved from Garage after creation.

For token rotation, first withdraw the UI route and stop the UI Deployment through the normal reviewed change. Revoke only the garage-webui token using /garage admin-token delete and its ID; remove only garage-webui-admin from Kubernetes, run webui-credentials, then restore the UI. Keep token values out of shell arguments, logs and local files. Preserve this Secret with the other runtime credential backups. UI scratch data needs no backup; objects remain in Garage and the UI stores no persistent user state with proxy authentication.

6. Rollback Strategy

To withdraw the administration UI, revert its Deployment, two Services, IngressRoute, NetworkPolicy, ConfigMap generator and image override through the reviewed Fleet change. Then revoke its dedicated token and remove only the garage-webui-admin Secret. Do not remove garage-rpc or application Secrets. The Garage StatefulSet, volumes and S3 endpoints continue independently.

To withdraw external S3 access, revert the IngressRoute, its Kustomize resource entry and the worker-IP NetworkPolicy rule through the usual reviewed Fleet change. This preserves internal Loki/Mimir access and storage.

Revert client endpoint/Secret references through a reviewed configuration change, keeping application PVCs and Garage volumes. The old external S3 store previously denied access: restoring its configuration alone does not restore service. Objects written to Garage stay there; moving them requires a separate migration. Stop application ingestion before any destructive storage operation. Do not delete Garage PVCs, node directories, RPC Secret or bucket contents as part of a configuration rollback. Avoid downgrading Garage across incompatible on-disk formats; use the upstream upgrade guidance and a verified backup.

7. Post-Incident Actions

Record health, node identities, layout version, S3 probes and Loki/Mimir readiness/ingestion results. Update the source, changelog and progress tracker. Arrange and test independent object-data and runtime-Secret backup coverage.