Skip to content

OpenFamily runbook

Metadata

Field Value
Service OpenFamily
Cluster / context homelab / homelab-k3s
Namespace openfamily
External endpoint https://openfamily.home.mutana.fr
Last reviewed 2026-09-14

Trigger Conditions

Use this runbook for initial onboarding, failed health probes, unavailable API or WebSocket traffic, database failures, stuck PVCs, backup/restore, upgrades, and rollback. Obtain explicit approval before any live-cluster mutation.

Preconditions

  • Use kube context homelab-k3s for direct operations.
  • Confirm kubectl config current-context before any mutation.
  • Ensure DNS for openfamily.home.mutana.fr reaches the homelab Traefik endpoint.
  • Bootstrap openfamily-secrets as described in openfamily/README.md.
  • Obtain explicit approval before kubectl apply, delete, rollout, or other live mutations.

1. Health Checks

kubectl kustomize openfamily/overlays/homelab
kubectl --context homelab-k3s -n openfamily get pods,svc,pvc,ingressroute
kubectl --context homelab-k3s -n openfamily get events --sort-by=.lastTimestamp
curl --fail --silent --show-error https://openfamily.home.mutana.fr/health

Expected steady state is one Ready client pod, one Ready server pod, one Ready PostgreSQL pod, two Bound PVCs, and an HTTP 200 health response.

First-use onboarding

  1. Open the external URL and create the first owner account.
  2. Test a second browser session and an invitation link.
  3. Add the Android or installed PWA client and confirm real-time updates.
  4. If VAPID keys were configured, grant notification permission and test Web Push.
  5. Disable server registration after onboarding if open registration is no longer wanted.

2. Troubleshooting Workflows

Check component logs without displaying Secret data:

kubectl --context homelab-k3s -n openfamily logs deployment/client --tail=200
kubectl --context homelab-k3s -n openfamily logs deployment/server --tail=200
kubectl --context homelab-k3s -n openfamily logs statefulset/postgres --tail=200
  • Pending pods: verify the expected Secret exists and both PVCs are Bound.
  • Server probe failures: inspect PostgreSQL readiness and server migration logs.
  • UI loads but API fails: confirm the Service is named server; the upstream client nginx config depends on that DNS name.
  • WebSocket failures: confirm Traefik routes the request to the client and that the client proxy receives the Upgrade headers.
  • Push failures: verify VAPID key pairing and VAPID_SUBJECT without printing secret values.

3. Disaster Recovery

Create a logical PostgreSQL dump with an approved backup workflow and back up the uploads PVC. Never paste the database password into a command or log. Verify that backups are non-empty and restorable before upgrades.

4. Scaling and Resource Management

Keep the server at one replica while it uses a ReadWriteOnce uploads PVC. Keep PostgreSQL at one replica. The stateless client can scale independently if needed. Review observed CPU and memory before changing requests or limits.

5. Maintenance Procedures

Before an application or database upgrade, create and verify a fresh logical database dump and uploads backup. Read upstream release notes, render the overlay locally, then deploy through the normal GitOps workflow.

6. Rollback Strategy

Revert the Git change to the previous image/manifests and let Fleet reconcile, or apply the previously validated overlay after approval. Do not delete either PVC during rollback. Database migrations may be forward-only, so take a fresh logical dump before changing OpenFamily versions.

7. Post-Incident Actions

Record the failure mode, recovery time, data-loss window, affected OpenFamily version, and backup used. Update probes, resource sizing, backup coverage, and this runbook when the incident reveals a repeatable gap.