Skip to content

vaultwarden Runbook

Metadata

Field Value
Service vaultwarden
Criticality Tier 1
Owner Platform / Security owner
Namespace vaultwarden
Clusters local
Last validated 2026-06-29
Related service page ../services/vaultwarden.md

Trigger Conditions

  • Vaultwarden UI or sync clients fail.
  • Websocket access breaks.
  • Users cannot unlock or sync vault items.
  • PVC-backed state or runtime secrets are unavailable.

1. Health Checks

kubectl -n vaultwarden get pods,svc,pvc,ingressroute
kubectl -n vaultwarden logs statefulset/vaultwarden --tail=200
# Readiness/liveness probes hit GET /alive on port 8080. Verify it returns 200:
kubectl -n vaultwarden exec statefulset/vaultwarden -- \
  wget -qO- http://localhost:8080/alive && echo

A pod that is not Ready is failing the /alive readiness probe and will not receive traffic.

2. Troubleshooting Workflows

Check ingress, websocket handling, and PVC health first.

kubectl -n vaultwarden describe statefulset vaultwarden
kubectl -n vaultwarden describe ingressroute
kubectl -n vaultwarden get secret
kubectl -n vaultwarden describe pod -l run=vaultwarden

Common causes:

  • Pod stuck NotReady / restarting: failing /alive probe, missing or malformed vaultwarden-secrets, or /data PVC not attached.
  • /tmp permission errors: the container runs with a read-only root filesystem; /tmp is an emptyDir. If vaultwarden-server cannot write, confirm the tmp volume is mounted.
  • SMTP errors: check SMTP_* values in ConfigMap vaultwarden-config and credentials in Secret vaultwarden-secrets.
  • Admin panel locked out: verify ADMIN_TOKEN in Secret vaultwarden-secrets.
  • Client fails with invalid type: JsValue(Object(...)), expected a string (e.g. bw get password, browser extension shows an empty vault): server/client version skew. Bitwarden clients v2026.7.0+ require Vaultwarden 1.37.0+; running an older server against a newer client fails cipher deserialization. Check the running image with kubectl -n vaultwarden get pod -o jsonpath='{.items[*].spec.containers[*].image}' — if it lags behind the pinned tag in base/vault-statefulset.yaml, the pod hasn't picked up a manifest bump yet and needs a rollout.
  • Pod exits immediately with LOG_LEVEL should follow the format info,vaultwarden::api::icons=debug, invalid: <value>: LOG_LEVEL in vault-config.env carries quotes, padding, or an inline comment. Kustomize's configMapGenerator takes everything after the first = literally, so LOG_LEVEL= "warn" # less noise becomes that whole string. Write the bare value (info, warn, ...) on its own line. Confirm with kubectl -n vaultwarden get cm vaultwarden-config -o yaml — every value must be bare.
  • A setting in vault-config.env appears to have no effect: same cause, quieter symptom. For boolean settings get_env_bool() returns None on an unparseable value and vaultwarden silently uses its compiled-in default, so e.g. SENDS_ALLOWED=false # unused leaves Sends enabled. String settings are used verbatim, so a malformed IP_HEADER silently breaks client-IP resolution and with it the per-client login and admin rate limits. Check the rendered ConfigMap rather than the env file.
  • A setting matches the ConfigMap but not the running behavior: values changed through the /admin panel are persisted to $DATA_FOLDER/config.json and override the environment. Vaultwarden logs [WARNING] The following environment variables are being overridden by the config.json file at startup with the affected names.
  • Pod stuck in CreateContainerConfigError with secret "vaultwarden-secrets" not found: the Secret is created out of band and nothing in Git generates it. Create it — see Disaster Recovery step 2 below.
  • Rancher Fleet shows ErrApplied on the vaultwarden bundle with loading KV pairs: env source files: [vault-secrets.env]: '.../vault-secrets.env' doesn't exist: a secretGenerator reading the gitignored .vault-secrets.env has been reintroduced on the build path. Fleet's clone never contains that file. There must be no secretGenerator anywhere in this workload; the Secret is created out of band. See vaultwarden/README.md.

3. Disaster Recovery

  1. Restore the data volume from snapshot.
  2. Recreate the runtime Secret. Populate overlays/local/.vault-secrets.env from vault-secrets.env.example (SMTP credentials + ADMIN_TOKEN), then:
kubectl create namespace vaultwarden --dry-run=client -o yaml | kubectl apply -f -
kubectl -n vaultwarden create secret generic vaultwarden-secrets \
  --from-env-file=vaultwarden/overlays/local/.vault-secrets.env \
  --dry-run=client -o yaml | kubectl apply -f -

Nothing in Git generates this Secret; it must exist before the workload reconciles. 3. Reconcile vaultwarden/overlays/local — kubectl apply -k locally, or let Fleet reconcile the same directory. 4. Validate /alive returns 200 and web login + client sync work.

4. Scaling and Resource Management

kubectl -n vaultwarden top pod

Resource changes are usually small, but adjust memory or storage in Git if the StatefulSet becomes constrained.

5. Maintenance Procedures

  • Rotate ADMIN_TOKEN and SMTP_PASSWORD in Secret vaultwarden-secrets.
  • After editing vault-config.env, check the rendered ConfigMap for bare values before applying: kubectl kustomize vaultwarden/overlays/local | yq 'select(.kind == "ConfigMap") | .data'. There is a single overlay, so this is exactly what Fleet will apply too.
  • Do not add a secretGenerator to overlays/local. It would break every Fleet sync — that is what forced the old duplicate overlay and the config drift behind it.
  • Validate websocket behavior after Traefik changes.
  • Schedule updates carefully because this service stores credentials.
  • Before a vaultwarden version upgrade or downgrade, snapshot the data PVC: data-format changes can make downgrades unsafe.

6. Rollback Strategy

  • Revert the overlay to the previous working revision in Git.
  • Restore the prior data snapshot if a configuration or version change corrupts startup.
  • A vaultwarden image downgrade after a data-format change may be unsafe; restore from a pre-change snapshot rather than rolling the image back in place.

7. Post-Incident Actions

  1. Add a changelog fragment for recovery work.
  2. Update the service page if exposure or secret handling changed.
  3. Extend this runbook with any newly discovered failure mode.