Skip to content

AppFlowy Cloud Runbook

Metadata

Field Value
Service appflowy
Criticality Tier 1
Owner Platform owner
Namespace appflowy
Clusters local
Last validated 2026-08-01
Related service page ../services/appflowy.md

Trigger Conditions

  • https://notes.mutana.fr is unavailable or returns errors.
  • A client reports synchronization, login, or collaboration failures.
  • appflowy-cloud, gotrue, admin-frontend, appflowy-web, appflowy-worker, or appflowy-nginx is not Ready.
  • The postgres, redis, or minio PVC is not Bound.
  • The appflowy-secrets Secret is unavailable or a workload is stuck on CreateContainerConfigError.

1. Health Checks

kubectl -n appflowy get deploy,statefulset,pod,svc,pvc,ingressroute
kubectl -n appflowy get events --sort-by='.lastTimestamp' | tail -n 30
kubectl -n appflowy logs deploy/appflowy-cloud --tail=200
kubectl -n appflowy logs deploy/gotrue --tail=150
kubectl -n appflowy logs deploy/appflowy-worker --tail=150
kubectl -n appflowy logs statefulset/postgres --tail=100
kubectl -n appflowy get endpoints appflowy-cloud gotrue appflowy-nginx

appflowy-cloud readiness gate: kubectl -n appflowy exec deploy/appflowy-cloud -- curl -fsS http://127.0.0.1:8000/api/health.

2. Troubleshooting Workflows

Check prerequisites and workload status without displaying Secret data:

kubectl -n appflowy get secret appflowy-secrets
kubectl -n appflowy describe deploy appflowy-cloud
kubectl -n appflowy describe statefulset postgres
kubectl -n appflowy describe pod -l app.kubernetes.io/part-of=appflowy

Common causes, in priority order:

  • A missing or mismatched appflowy-secrets Secret (especially a GOTRUE_JWT_SECRET that differs between gotrue and cloud, or a POSTGRES_* mismatch) shows up as CreateContainerConfigError or auth failures.
  • An unbound local-path PVC blocking postgres, redis, or minio.
  • PostgreSQL startup delay; appflowy-cloud and gotrue retry the connection on boot, so transient errors clear once postgres is Ready.
  • The bundled appflowy-nginx failing to start, which breaks all external access even when backends are healthy — check deploy/appflowy-nginx.
  • Traefik routing or TLS failure at notes.mutana.fr; confirm DNS resolves to the cluster edge.
  • Missing verification or recovery mail; check GoTrue logs for SMTP transport errors and confirm that SMTP_PASSWORD exists in appflowy-secrets without displaying its value.

3. Disaster Recovery

  1. Take a logical PostgreSQL dump using credentials from the secure operator environment; do not place credentials or dumps in Git.
  2. Mirror the MinIO bucket (attachments) and preserve a secure copy of appflowy-secrets. Optionally snapshot the PVCs.
  3. For recovery, restore the Secret first, then the PostgreSQL dump into a compatible pgvector instance, then the MinIO data.
  4. Reconcile appflowy/overlays/local and wait for all workloads to become Ready.
  5. Validate browser login at /console and synchronization from a test client.

Record the first successful restore drill and the measured RPO/RTO in this runbook.

4. Scaling and Resource Management

Every workload runs one replica. Resource requests are conservative starting points; inspect actual usage before changing them:

kubectl -n appflowy top pod
kubectl -n appflowy top pod --containers

appflowy-cloud and PostgreSQL are the most likely to need more memory as data grows. PostgreSQL connection pool sizing (APPFLOWY_DATABASE_MAX_CONNECTIONS=40) is documented in the overlay ConfigMap.

5. Maintenance Procedures

  • Back up PostgreSQL and MinIO before every AppFlowy upgrade.
  • Review upstream AppFlowy-Cloud release notes before changing pinned images; note that appflowy_web and the core services are versioned independently.
  • Rotate database, MinIO, SMTP, and JWT credentials only with a coordinated Secret update across all consuming workloads.
  • Keep AI/search disabled until their migrations have been tested against a disposable restore of the production database.

6. Rollback Strategy

Revert the GitOps manifests but preserve the PostgreSQL, MinIO, and Redis PVCs and the appflowy-secrets Secret. If an upgrade applied incompatible database migrations, restore the pre-upgrade dump rather than starting an older image against the migrated database.

7. Post-Incident Actions

  1. Record the cause, recovery steps, and any data-loss window.
  2. Update the measured RPO/RTO after any restore.
  3. Add a changelog fragment for recovery or security-relevant changes.
  4. Update this runbook when an incident reveals a missing diagnostic or step.