Skip to content

windmill Runbook

Metadata

Field Value
Service windmill
Criticality Tier 1
Owner Platform owner
Namespace windmill
Clusters local
Last validated 2026-07-23
Related service page ../services/windmill.md

Trigger Conditions

  • Windmill UI or API at https://windmill.mutana.fr fails.
  • CLI or API-token clients cannot authenticate to Windmill.
  • Windmill server, worker, LSP, or Postgres pods are not Ready.
  • Jobs are not progressing (queued jobs never move to a worker).
  • PVC-backed state or the windmill-secrets Secret is unavailable.

1. Health Checks

kubectl -n windmill get deploy,statefulset,pod,svc,pvc,ingressroute
kubectl -n windmill get events --sort-by='.lastTimestamp' | tail -n 20
kubectl -n windmill logs deploy/windmill-server --tail=200
kubectl -n windmill logs deploy/windmill-worker-default --tail=200
kubectl -n windmill logs deploy/windmill-worker-native --tail=100
kubectl -n windmill logs statefulset/windmill-postgres --tail=100

Verify the server health endpoint from inside the cluster:

kubectl -n windmill exec deploy/windmill-server -- \
  wget -qO- http://localhost:8000/ && echo

Confirm all pods are scheduled on the expected node:

kubectl -n windmill get pod -o wide

All Windmill pods should report NODE=layer7-vps1. If a pod is Pending with an unschedulable affinity error, the node is down or renamed.

2. Troubleshooting Workflows

Check the Secret and Postgres readiness before changing manifests.

kubectl -n windmill get secret windmill-secrets
kubectl -n windmill describe deploy windmill-server
kubectl -n windmill describe deploy windmill-worker-default
kubectl -n windmill describe statefulset windmill-postgres
kubectl -n windmill describe pod -l app.kubernetes.io/part-of=windmill

Common causes:

  • Missing Secret: create windmill-secrets with the documented required keys (POSTGRES_USER, POSTGRES_PASSWORD, POSTGRES_DB).
  • Server stuck in startup / role 'windmill_admin' does not exist: the initial database migration failed, usually because Postgres credentials were wrong or Postgres was not Ready when the server first booted. Fix the Secret, ensure Postgres is Ready, then restart the server: kubectl -n windmill rollout restart deploy/windmill-server.
  • DATABASE_URL malformed / connection refused / password authentication failed for user "$(POSTGRES_USER)": the $(...) was not expanded. This happens when DATABASE_URL is ordered before the POSTGRES_* secretKeyRefs in the container env list — Kubernetes only expands $(VAR) using vars defined earlier in the same list. Keep DATABASE_URL last. If instead the password itself is wrong, POSTGRES_PASSWORD is not URI-safe or mismatched; use a URI-safe password or pre-encode the connection string. Do not print the secret value while diagnosing.
  • MODE=indexer panics with "tantivy is not included in this binary/image": full-text search is Enterprise-Edition only. The CE image cannot run an indexer. Do not add an indexer Deployment on the CE image; the server falls back to database search. Only add one with the EE image and a valid licence.
  • Jobs stuck in queued state: the default worker is not running or not privileged. Confirm windmill-worker-default pod phase is Running and its container is privileged: true (required for subprocess jobs). Native-only jobs still run on windmill-worker-native.
  • windmill-worker-* pod shows ContainersReady=False but logs are healthy: the worker has no HTTP health endpoint on Community Edition, so it must not define an HTTP readinessProbe. The /ready endpoint is EE-only (the upstream Helm chart gates the probe behind enterprise.enabled). On CE a worker readinessProbe against /ready will fail forever. Remove the probe; readiness is "process running + logs healthy". Do not re-add an HTTP probe to CE workers.
  • Pod Pending with affinity error: layer7-vps1 is unavailable or was renamed. This is single-node scheduling by design; restore the node or relax the affinity in Git as a deliberate change.
  • Postgres not Ready: check PVC binding on layer7-vps1, PGDATA, and pg_isready probe failures.
  • Ingress failure: confirm the IngressRoute hostname, Traefik route, TLS, and the windmill-server service endpoint.

3. Disaster Recovery

  1. Confirm whether the incident is app-only, worker-only, storage-related, or cluster-wide.
  2. Restore the Postgres PVC from the selected snapshot onto layer7-vps1.
  3. Recreate windmill-secrets from the secure out-of-band source.
  4. Reconcile windmill/overlays/local.
  5. Validate pod readiness, the server health endpoint, UI login, and a known test job run.

The restore drill for this service still needs to be validated. Treat the first real restore as a high-attention operation and capture the result in this runbook.

4. Scaling and Resource Management

kubectl -n windmill top pod
kubectl -n windmill top pod --containers
kubectl -n windmill describe deploy windmill-server
kubectl -n windmill describe deploy windmill-worker-default

The deployment intentionally runs one replica per component for a low footprint. Scaling and resource changes should be made in Git after checking Postgres capacity and the layer7-vps1 node resources. To run only in-process native jobs, scale windmill-worker-default to 0 (the native worker stays unprivileged).

5. Maintenance Procedures

  • Rotate Postgres credentials only with an explicit rotation plan and a tested rollback path; update windmill-secrets and restart the server and workers.
  • Snapshot Postgres before Windmill version upgrades.
  • Re-check upstream release notes before changing the pinned Windmill image tag; worker security or job-isolation behaviour can change between versions.
  • The default worker is privileged by design. Do not remove privileged: true from the default worker unless you also disable subprocess job types — doing so will silently break Python/Deno/bash jobs.

6. Rollback Strategy

  • Revert the Git changes that add the windmill/ workload and remove the Fleet path from fleet/layer7/gitrepo-main.yaml.
  • If the workload already ran, preserve the Postgres PVC snapshot and windmill-secrets before deleting resources.
  • A downgrade after application data changes may require restoring a pre-change Postgres snapshot rather than rolling the image back in place, because the server runs forward-only migrations on boot.

7. Post-Incident Actions

  1. Add a changelog fragment for any recovery or security-relevant intervention.
  2. Update the service page if dependencies, exposure, or secret handling changed.
  3. Update this runbook with commands or checks learned during the incident.
  4. Record backup and restore validation results in context/progress-tracker.md.