Grounded Runbook¶

Metadata¶

Field	Value
Service	grounded
Criticality	TIER_3
Owner	arnaud
Cluster / namespace	local / grounded
Last validated	2026-08-27 (rendered manifests; not live-deployed)
Related service page	docs/services/grounded.md

Trigger Conditions¶

Use this runbook when a Grounded workload is not Ready, scraping/search fails, the management UI is unavailable, an MCP client cannot connect through the tunnel, storage is at risk, or a backup/restore/upgrade is planned.

Production readiness gates¶

Do not call the service production-ready until all four gates are closed:

grounded-secrets exists with working OPENAI_API_KEY and OPENAI_API_BASE values.
A backup has been copied off the local-path node and restored successfully; retention, RPO, and RTO are written down.
Client access uses the accepted Kubernetes tunnel, or native MCP OAuth/OIDC is designed and tested. Do not remove Authelia from the public route.
Grounded is added to the authoritative live Fleet GitRepo. The tracked fleet/layer7/gitrepo-main.yaml does not currently match the live split GitRepo topology and must not be applied as a shortcut.

The namespace was absent from the local cluster during the 2026-08-27 review. The following checks and deployment steps are therefore a preflight, not a record that production has already been deployed.

1. Health Checks¶

Preflight and deployment¶

Confirm the intended context before every command:

kubectl config current-context
kubectl --context oci cluster-info
kubectl --context oci get namespace grounded

Inspect live Fleet ownership. This is read-only:

kubectl --context rancher get gitrepos -A
kubectl --context rancher -n fleet-local get gitrepo local -o yaml

Create the namespace and Secret only after the provider endpoint and key are known. The endpoint variable name is OPENAI_API_BASE:

kubectl --context oci create namespace grounded \
  --dry-run=client -o yaml | kubectl --context oci apply -f -
cp grounded/overlays/local/grounded-secrets.env.example \
  grounded/overlays/local/grounded-secrets.env
# Replace both placeholders, then create the Secret without printing its data.
kubectl --context oci -n grounded create secret generic grounded-secrets \
  --from-env-file=grounded/overlays/local/grounded-secrets.env \
  --dry-run=client -o yaml | kubectl --context oci apply -f -

Verify only the key names, never their values:

kubectl --context oci -n grounded get secret grounded-secrets \
  -o go-template='{{range $key, $_ := .data}}{{printf "%s\n" $key}}{{end}}'

Render before changing the cluster:

kubectl kustomize grounded/overlays/local > /tmp/grounded-rendered.yaml
kubectl --context oci apply --dry-run=client -f /tmp/grounded-rendered.yaml

Prefer the corrected Fleet source of truth. If a manual deployment is explicitly approved after all gates close:

kubectl --context oci apply -k grounded/overlays/local
kubectl --context oci -n grounded rollout status \
  statefulset/grounded-worker --timeout=5m
kubectl --context oci -n grounded rollout status \
  deployment/grounded-mcp --timeout=5m
kubectl --context oci -n grounded rollout status \
  deployment/grounded-web --timeout=5m

Runtime health checks¶

kubectl --context oci -n grounded get \
  statefulset,deployment,pod,service,pvc,ingressroute,networkpolicy
kubectl --context oci -n grounded get pods -o wide
kubectl --context oci -n grounded get events \
  --sort-by='.lastTimestamp' | tail -n 30

Expected steady state:

worker StatefulSet 1/1, MCP Deployment 1/1, web Deployment 1/1;
all three pods Ready with no restart loop;
data-grounded-worker-0 PVC Bound;
worker port 8080 has no external IngressRoute;
worker logs show the configured embedding model without provider errors.

Inspect a failing component:

kubectl --context oci -n grounded describe pod <pod-name>
kubectl --context oci -n grounded logs <pod-name> \
  --all-containers --tail=200
kubectl --context oci -n grounded logs <pod-name> \
  --all-containers --previous --tail=200

Verify client and web access¶

Open the client tunnel on the machine where Codex, Claude Code, VS Code, or ZCode runs:

kubectl --context oci -n grounded port-forward \
  service/grounded-mcp 6280:6280

Point the client at http://127.0.0.1:6280/mcp. Then ask it to call list_libraries and perform one search_docs request. A successful TCP connection alone does not prove that worker search or the embedding provider is working.

Verify the management UI separately in a browser at https://grounded.mutana.fr. The browser should complete the Authelia flow.

Do not test a headless client against https://mcp-docs.mutana.fr/mcp and interpret the forward-auth redirect as a Grounded failure. That route is reserved until native MCP authentication is implemented.

2. Troubleshooting Workflows¶

Provider, scrape, or search errors¶

Check worker logs first. Only the worker receives provider credentials:

kubectl --context oci -n grounded logs statefulset/grounded-worker \
  --tail=300

401 or 403: rotate or correct OPENAI_API_KEY.
endpoint or connection failure: verify OPENAI_API_BASE; the obsolete name OPENAI_BASE_URL is ignored by Grounded v2.4.5.
dimension/model mismatch after a configuration change: restore the previous model or plan a full re-scrape to rebuild vectors.
a URL resolving to a private, loopback, link-local, or cluster address should be rejected by design. Do not disable the SSRF protection; copy approved internal material to a safe public source or design a narrow allow-list in a separate reviewed change.

After rotating the Secret, restart only the worker:

kubectl --context oci -n grounded rollout restart \
  statefulset/grounded-worker

Pod pending or PVC unavailable¶

kubectl --context oci -n grounded describe pod grounded-worker-0
kubectl --context oci -n grounded get pvc data-grounded-worker-0 -o wide
kubectl --context oci -n grounded describe pvc data-grounded-worker-0
kubectl --context oci get pv
kubectl --context oci get nodes -o wide

local-path data is tied to the selected node. A rescheduled pod cannot use the data if that node or its local volume path is unavailable. Do not delete the PVC while investigating.

OOMKilled or slow startup¶

kubectl --context oci -n grounded top pod --containers
kubectl --context oci -n grounded describe pod <pod-name>

Memory requests/limits are worker 1Gi/2Gi and MCP/web 256Mi/512Mi. Correlate a limit increase with library size and observed peak usage; change it in Git rather than leaving a live patch.

Ingress or Authelia failure¶

kubectl --context oci -n grounded get ingressroute,service,endpoints
kubectl --context oci -n grounded describe ingressroute grounded-web
kubectl --context oci -n authelia logs deployment/authelia --tail=200

An Authelia outage blocks the public web UI. The Kubernetes MCP tunnel bypasses Traefik and Authelia but still requires authorized cluster access.

3. Disaster Recovery¶

Backup contract¶

No recurring Grounded backup or VolumeSnapshot is configured. local-path does not provide a repository-backed snapshot guarantee. A production backup must:

leave the node and cluster;
include the complete /data volume, not only documents.db;
run while the worker is stopped, or use a SQLite-aware online backup tool;
have a defined schedule and retention;
be restored and searched in a drill.

Do not copy only a live documents.db: SQLite runs in WAL mode and committed data may still be in companion WAL files.

Safe offline backup sequence¶

The destination and backup implementation are intentionally left open until the operator selects them. Once available:

Stop new scrape/delete operations and wait for active jobs to finish.
Record the worker replica count and PVC/PV identity.
Scale grounded-worker to zero and wait for its pod to terminate.
With the approved node/PV backup mechanism, copy the entire PVC data directory to the off-cluster destination.
Record checksum, timestamp, image version, and embedding model with the backup metadata. Do not record provider secrets.
Scale the worker back to one and verify a Grounded search.

The MCP and web gateways may remain running during the offline copy, but search requests will fail until the worker returns.

Restore sequence¶

Scale the worker to zero and confirm no worker pod remains.
Preserve the damaged data directory for forensics when possible.
Restore the entire backed-up data directory to the PVC, retaining ownership compatible with UID/GID 1000.
Restore the matching image/configuration revision and verify grounded-secrets exists.
Scale the worker to one; inspect migration/startup logs.
Verify list_libraries, keyword search, vector search, and one source URL.
Record measured RPO and RTO.

If no backup exists, recreate the PVC only with explicit approval, then re-add and re-scrape every library from its source. The indexed documentation is reconstructible; the curated list, versions, and crawl choices still need an operator record.

4. Scaling and Resource Management¶

Keep grounded-worker at exactly one replica because it owns the SQLite writer and in-memory scrape queue. MCP and web are stateless, but scale them only after testing HTTP MCP session behavior and routing. Review observed usage with:

kubectl --context oci -n grounded top pod --containers

Commit resource changes to Git and reconcile them through Fleet. Avoid leaving an emergency live resize as configuration drift.

5. Maintenance Procedures¶

Image upgrade: read upstream release notes and store-migration notes, change the pinned newTag, render, deploy, and watch worker logs first.
Library maintenance: use the web UI for deliberate add/refresh/remove operations. Deletion affects every client.
Embedding model change: take a verified backup, update the model/provider, and plan a full vector re-scrape. This is not a routine rolling change.
Scaling: keep the worker at exactly one replica. Scale stateless gateways only after testing MCP session behavior.

6. Rollback Strategy¶

For manifest rollback, revert the change in Git and reconcile the same Fleet bundle. For an approved manual rollback:

kubectl --context oci apply -k grounded/overlays/local

A binary downgrade may not read a newer database schema. Restore a backup taken with the matching image if startup or migration fails; never start two worker versions against the same PVC.

7. Post-Incident Actions¶

Update this runbook, the service page, and context/progress-tracker.md with:

root cause and affected libraries;
backup timestamp, measured RPO/RTO, and restore result;
any changed provider, model, resource, or access assumption;
a changelog fragment for a manifest, security, or recovery change.