Skip to content

forgejo Runbook

Metadata

Field Value
Service forgejo
Criticality Tier 2
Owner Platform / SCM owner
Namespace forgejo
Clusters local, ozilab
Last validated 2026-09-02
Related service page ../services/forgejo.md

Trigger Conditions

  • Forgejo UI or Git traffic is unavailable on git.mutana.fr (local) or git.lab.mutana.fr (ozilab).
  • Repository operations fail.
  • PostgreSQL becomes unhealthy.
  • Ingress or authentication middleware breaks the route.
  • Pods report CreateContainerConfigError (missing Secrets).

1. Health Checks

kubectl -n forgejo get pods,svc,pvc,ingressroute
kubectl -n forgejo logs deploy/forgejo --tail=200
kubectl -n forgejo get statefulset

On ozilab, inspect the Fleet bundle state as well: kubectl -n fleet-default get bundle | grep forgejo.

2. Troubleshooting Workflows

kubectl -n forgejo describe deploy forgejo
kubectl -n forgejo logs statefulset/forgejo-db --tail=100
kubectl -n forgejo describe ingressroute
kubectl -n forgejo get secret forgejo-secrets

Check in order: the out-of-band Secret exists with all four keys, DB auth, PVC attachment, Traefik middleware references. A pod stuck in CreateContainerConfigError means forgejo-secrets is missing — run forgejo/bootstrap-secrets.sh overlays/<cluster>.

3. Disaster Recovery

  1. Restore PostgreSQL state from backup or PVC snapshot. The postgres-data volume mounts at /var/lib/postgresql and the cluster lives in the versioned subdirectory (<major>/docker) — restore must target that path, not the volume root (postgres 18+ layout, docker-library/postgres#1259).
  2. Ensure the out-of-band Secrets exist (re-run forgejo/bootstrap-secrets.sh overlays/<cluster>; values come from the operator vault, not Git).
  3. Reconcile forgejo/overlays/<cluster> (kubectl apply, or let the ozilab GitRepo sync).
  4. Validate UI and Git clone/push paths on the cluster's hostname.

Locked out of ozilab Forgejo

Registration is disabled and the web installer is locked, so there is no self-service recovery. Symptom: no account works, or the admin password is lost.

# 1. Confirm the pod is serving.
kubectl -n forgejo get deploy forgejo
kubectl -n forgejo exec deploy/forgejo -- forgejo admin user list

# 2. Reset an existing account's password.
kubectl -n forgejo exec -it deploy/forgejo -- \
  forgejo admin user change-password --username <user> --password '<new>' \
    --must-change-password=true

# 3. Or create a fresh admin.
kubectl -n forgejo exec -it deploy/forgejo -- \
  forgejo admin user create --admin --username <user> --email <mail> \
    --random-password

If the binary refuses to run as root, re-run the command through su git -c. overlays/ozilab/bootstrap-admin.sh automates both paths and is idempotent: re-running with an existing username resets that account's password.

Do not re-enable registration as a recovery step. It reopens the instance to anyone who can reach it, which is the condition this configuration exists to prevent.

Verifying access control is actually applied

Config keys written without a section segment are silently ignored — this had happened to REQUIRE_SIGNIN_VIEW before 2026-09-22. Check the rendered app.ini, not the env file:

kubectl -n forgejo exec deploy/forgejo -- \
  sh -c 'sed -n "/^\[service\]/,/^\[/p" /data/gitea/conf/app.ini'
# Expect DISABLE_REGISTRATION = true and REQUIRE_SIGNIN_VIEW = true.
# A stray [REQUIRE] section means a key lost its section again.

curl -sS -o /dev/null -w '%{http_code}\n' https://git.lab.mutana.fr/explore/repos
# Expect a redirect to /user/login, not 200.

4. Scaling and Resource Management

kubectl -n forgejo top pod
kubectl -n forgejo describe deploy forgejo

The app requests 250m/512Mi with a 2Gi memory limit; the database requests 100m/256Mi with a 1Gi limit. Adjust in Git if repository traffic exceeds these. Storage sizes live in the overlay PVC (10Gi app) and the StatefulSet volumeClaimTemplates (5Gi db — immutable in place; resizing is a planned migration).

5. Maintenance Procedures

  • Rotate mailer or DB credentials: update the overlay's gitignored env file, re-run bootstrap-secrets.sh overlays/<cluster>, restart pods.
  • Validate ingress and auth middleware after edge changes.
  • Recheck PVC ownership before node maintenance (local-path volumes are node-pinned; app and db pods are co-scheduled via pod affinity).
  • Image upgrades: bump the pinned tags in overlays/<cluster>/kustomization.yaml only (never base). Forgejo runs migrations at startup — the startup probe tolerates several minutes.

6. Rollback Strategy

  • Revert the overlay to the previous working revision.
  • Restore the DB snapshot if application schema or config changes failed.

7. Post-Incident Actions

  1. Add a changelog fragment for manual recovery.
  2. Update the Forgejo service page if topology or access changed.
  3. Extend this runbook if a new operational hazard was identified.