Garage

Metadata

Field	Value
Purpose	S3 object storage for Loki and Mimir
Cluster / namespace	cc-obs-kube / garage
Last reviewed	2026-10-24

1. Service Overview

Garage replaces the inaccessible external S3 backend with fresh buckets approved by the owner. It stores three copies on separate Jelastic worker disks.

2. Architecture Diagram

Loki (loki, loki-ruler, loki-admin) --+
                                   +--> garage:3900 --> garage-{0,1,2}
Mimir (mimir, mimir-tsdb) -----------+                   one per worker

Browser --> Traefik + Authelia --> garage-webui:3909
                                   +--> garage-admin:3903 (admin API)
                                   +--> garage:3900       (objects)

3. Deployment Specifications

garage/fleet.yaml targets cc-obs-kube with overlays/jls. Native Kubernetes resources run dxflrs/garage:v2.4.1: three StatefulSet replicas, required worker anti-affinity, six static retained local PVs, and garage-local StorageClass. Each worker supplies 50 GiB data and 1 GiB metadata, giving approximately 50 GiB usable object capacity with replication factor 3. Local directories share the worker root disk; these sizes are placement allocations, not filesystem quotas. The PDB requires two available pods. Requests are 100m CPU / 512Mi per pod; limits are 1000m / 1Gi, starting from the upstream small-deployment guidance.

The stateless administration Deployment runs khairul169/garage-webui:1.1.0 with one replica. It initially uses the same requests/limits as a Garage pod; measure browsing/upload use before tuning. It has HTTP readiness/liveness checks, a read-only root filesystem, an ephemeral upload scratch volume and no PVC. The UI NetworkPolicy allows ingress-worker IPs on port 3909 only.

4. Configuration Guide

garage/overlays/jls/garage.toml selects SQLite, consistent quorum operation, region garage, and stable peer DNS with public node IDs. RPC addresses use the existing Weave pod subnet. The RPC secret is injected through GARAGE_RPC_SECRET from garage-rpc; no admin master token is configured. CLI administration uses internal authenticated RPC. Bootstrap scripts hold generated values in memory and create application Secrets directly through kubectl stdin.

bootstrap.py webui-credentials creates a dedicated, non-expiring dynamic admin token with full scope in garage/garage-webui-admin (admin-token). The UI receives it only as API_ADMIN_KEY; its separate browser-visible webui.toml contains no credentials. Existing valid credentials are reused; a missing Secret with an existing named token requires operator recovery. No Garage master token or StatefulSet configuration change is required.

5. Access Protocols

Internal S3: http://garage.garage.svc.cluster.local:3900 for Loki and Mimir. External S3: https://s3.mutana.site for authorized S3 clients, using region garage, SigV4 and path-style requests. The garage IngressRoute uses websecure, preserves the Host header and path, and forwards to port 3900 with the existing default-security-headers middleware. Authentication uses Garage keys directly; browser SSO and basic auth would interfere with S3 clients. TLS uses Traefik's default Cloudflare Origin certificate, with DNS routed through the cluster's existing Cloudflare-proxied ingress.

NetworkPolicy allows S3 ingress from garage, loki and mimir namespaces, plus the three host-network Traefik worker IPs (172.16.8.168, 172.16.8.170, 172.16.8.174). These addresses permit node-originated S3 traffic and must be updated when ingress workers change. RPC and admin traffic remain restricted to garage. The admin API is reachable through the private garage-admin ClusterIP Service and has no direct public IngressRoute. Application keys grant read/write on their own buckets, without bucket creation or owner permissions.

Administration UI: https://garage.mutana.site, Traefik websecure with authelia@file and security headers. The existing wildcard Authelia policy requires external users to complete 2FA; its trusted-network and OPTIONS bypass rules also apply. Admitted users have full Garage administrative access via the UI's backend proxy, including application objects and key management. The browser uses the UI origin for API/object operations; only the UI backend connects to the private admin and S3 Services.

6. Operations and Observability

The readiness probe checks /health; TCP RPC probes check process liveness. bootstrap.py health must report healthy, three storage nodes up, and all 256 partitions OK. verify-s3.py verifies CRUD across peers and credential isolation. Watch worker free disk, resync queues, pod restarts, application upload failures and snapshot growth. See the runbook.

7. Backup and Recovery Notes

PVCs, PVs and namespace are retained. Worker-local data survives pod replacement on the same worker, but does not automatically move to a replacement worker. Automatic six-hour metadata snapshots stay on that disk. They are not an off-cluster backup. Preserve runtime Secrets securely using the operator's backup process; no independent backup destination was supplied for this rollout.

8. Release and Change Notes

2026-10-24: add the Authelia-protected administration UI at garage.mutana.site, private admin Service and out-of-band dynamic-token bootstrap.

2026-10-24: add Traefik HTTPS S3 access at s3.mutana.site and S3-only NetworkPolicy allowances for the host-network ingress workers.

2026-10-20: deploy Garage and switch the jls Loki/Mimir clients to fresh internal S3 buckets. Existing application PVCs and old credential generators are retained for an explicit rollback; reverting configuration does not migrate new objects.