Skip to content

defectdojo Runbook

Metadata

Field Value
Service defectdojo
Criticality Tier 2
Owner Platform / Security owner
Namespace defectdojo
Clusters homelab, local
Last validated 2026-07-02
Related service page ../services/defectdojo.md

Trigger Conditions

  • DefectDojo UI is unavailable.
  • Login fails for the bootstrap admin on a fresh install.
  • The initializer Job fails or stays blocked.
  • PostgreSQL, Valkey, or the Django pod becomes unhealthy.
  • Fleet reports must specify --enable-helm while applying DefectDojo.

1. Health Checks

kubectl -n defectdojo get pods,job,svc,pvc,ingressroute
kubectl -n defectdojo logs job/defectdojo-initializer --tail=200
kubectl -n defectdojo logs deploy/defectdojo-django -c uwsgi --tail=200
kubectl -n defectdojo logs deploy/defectdojo-django -c nginx --tail=200

Check that the initializer Job completed successfully before debugging the login flow.

Cluster-targeted commands

Cluster Context flag
homelab --context <homelab-context> or run from a shell kubeconfig pointing at homelab
local --context <local-context> or run from the Rancher-managed kubeconfig

For the local cluster, also verify node placement (all pods must be on the same node.io/defectdojo=true node):

# Confirm a node carries the label.
kubectl get nodes -l node.io/defectdojo=true
# Confirm every DefectDojo pod scheduled onto that node.
kubectl -n defectdojo get pods -o wide

Verify the API endpoint that CI/automation uses (the /api route bypasses Authelia, so a token-authenticated request should succeed):

curl -sS -H "Authorization: Token ${DD_API_TOKEN}" \
  https://defectdojo.mutana.fr/api/v2/userinfo/ | head

2. Troubleshooting Workflows

Bootstrap admin password does not work on first startup

  1. Confirm the expected password stored in the cluster Secret.
kubectl -n defectdojo get secret defectdojo \
  -o jsonpath='{.data.DD_ADMIN_PASSWORD}' | base64 -d && echo
  1. Inspect the initializer logs.
kubectl -n defectdojo logs job/defectdojo-initializer --tail=200

  1. If the initializer is stuck in CreateContainerConfigError because DD_ADMIN_PASSWORD is missing, fix the local secret input and re-reconcile the overlay. This is the intended fail-fast behavior.

  2. If the initializer already completed during an earlier broken bootstrap and the dojo-admin password still fails, reset it from the uwsgi container.

kubectl -n defectdojo exec -it deploy/defectdojo-django -c uwsgi -- \
  python manage.py changepassword dojo-admin
  1. If dojo-admin does not exist or bootstrap created the wrong account state, create a new superuser.
kubectl -n defectdojo exec -it deploy/defectdojo-django -c uwsgi -- \
  python manage.py createsuperuser
  1. After manual recovery, sign in through the UI and record which account is now the primary emergency admin.

General runtime diagnostics

kubectl -n defectdojo describe job defectdojo-initializer
kubectl -n defectdojo describe deploy defectdojo-django
kubectl -n defectdojo logs deploy/defectdojo-celery-worker --tail=100
kubectl -n defectdojo logs statefulset/defectdojo-postgresql --tail=100
kubectl -n defectdojo logs statefulset/defectdojo-valkey --tail=100

Look first for missing secrets, DB auth failures, CSRF or host configuration issues, and uwsgi restarts caused by memory pressure.

Fleet fails with must specify --enable-helm

Fleet inflates the DefectDojo chart itself. Its post-render Kustomize path must not be the local entry point, because that file contains a helmCharts: block for manual rendering. Verify both supported render paths:

# Fleet post-render resources: no --enable-helm
kubectl kustomize defectdojo/overlays/local/fleet
kubectl kustomize defectdojo/overlays/homelab/fleet

# Local/manual entry point: Helm explicitly enabled
kubectl kustomize defectdojo/overlays/local --enable-helm
kubectl kustomize defectdojo/overlays/homelab --enable-helm

Confirm that defectdojo/fleet.yaml selects overlays/<cluster>/fleet for each targetCustomization. Do not add helmCharts: to that Fleet post-render directory. Before retrying the initial Fleet reconciliation, ensure the defectdojo and defectdojo-postgresql-specific Secrets already exist in the target namespace.

Fleet fails with PASSWORDS ERROR / global.postgresql.auth.password must not be empty

Root cause: Fleet renders Helm charts in a dry-run context where Helm's lookup() function is a no-op. The overlay intentionally leaves postgresql.auth.password empty and supplies credentials through postgresql.auth.existingSecret: defectdojo-postgresql-specific, so the Bitnami PostgreSQL subchart's upgrade-password guard calls lookup() at render time to confirm the password is recoverable. A Fleet reconciliation is a Helm upgrade (Release.IsUpgrade true); because Fleet's render-time lookup() cannot see any Secret — even one that exists on the target cluster — the guard fails fast on every resync.

This is why kubectl kustomize defectdojo/overlays/local --enable-helm, make validate, and helm template all stay green: they render without cluster access (install path), where lookup() is skipped and a password is auto-generated. Only helm upgrade --install against a cluster triggers the guard. Reproduce locally with the exact Fleet context:

helm template defectdojo ./defectdojo/charts/defectdojo-1.9.37/defectdojo \
  -f defectdojo/overlays/local/values.yaml -n defectdojo --is-upgrade
# EXIT=1, identical PASSWORDS ERROR

Resolution: inject postgresql.auth.password as a Helm value at install time via Fleet helm.valuesFrom (see defectdojo/fleet.yaml). When the value is present, the chart skips lookup() entirely. postgresql.auth.existingSecret still wins at pod runtime, so the pod continues to source its password from the defectdojo-postgresql-specific Secret via secretKeyRef. The injected value only satisfies the render-time guard.

Operator prerequisite (run against the management cluster). Use the same password already in the target cluster's runtime Secret so render-time and runtime values match:

# 1. Read the real password from the downstream cluster's runtime Secret:
PG_PWD=$(kubectl --context <cluster-context> -n defectdojo get secret \
  defectdojo-postgresql-specific -o jsonpath='{.data.postgresql-password}' | base64 -d)

# 2. Create the Fleet-values Secret on the MANAGEMENT cluster (fleet-default ns):
kubectl --context <management-context> -n fleet-default create secret generic \
  defectdojo-helm-values-<cluster> \
  --from-literal=values.yaml="$(printf 'postgresql:\n  auth:\n    password: "%s"\n' "$PG_PWD")" \
  --dry-run=client -o yaml | kubectl --context <management-context> apply -f -

# 3. Force a Fleet re-sync (the valuesFrom Secret is also covered by the caveat below).

Confirm the Secret exists on the management cluster:

kubectl --context <management-context> -n fleet-default get secret \
  defectdojo-helm-values-<cluster>

Caveat: Fleet does not auto-detect changes to valuesFrom Secrets (rancher/fleet#2085). After rotating the password, force a Fleet re-sync so the new value is read during the next install. Fleet reads the Secret from the namespace specified in the secretKeyRef on the management cluster, not from the downstream cluster's defectdojo namespace.

If the valuesFrom Secret exists and Fleet still fails, verify: the Secret is in fleet-default on the management cluster (not the downstream cluster); the key is literally values.yaml; the value parses as YAML (postgresql.auth.password is a string, so quote it); and the secretKeyRef.name in defectdojo/fleet.yaml matches the Secret name (defectdojo-helm-values-homelab or defectdojo-helm-values-local).

Pods stay Pending on the local cluster (node placement)

All DefectDojo pods carry nodeSelector: node.io/defectdojo=true. If pods are Pending with FailedScheduling, exactly one of these is true:

  1. No node is labelled. Fix: 
    kubectl label node <chosen-node> node.io/defectdojo=true --overwrite
  2. The chosen node is cordoned, drained, or out of capacity. Either pick another node to label, or add capacity, then label the new node.
  3. local-path PVCs cannot bind because volumeBindingMode: WaitForFirstConsumer is waiting for a pod that cannot schedule — this resolves automatically once the label is applied.

Confirm with kubectl -n defectdojo describe pod <pod> and read the events.

Authelia blocks API / CI imports

CI and the Nessus importer call https://defectdojo.mutana.fr/api/v2/.... If imports return 401/403 with an Authelia-style body (HTML login page) instead of a DefectDojo JSON error, the /api IngressRoute route is missing or has lower priority than the UI route. Verify the IngressRoute renders two routes and the /api route has the higher priority:

kubectl -n defectdojo get ingressroute defectdojo -o yaml | grep -A2 priority

Expected: the PathPrefix('/api') route has priority: 100 and no authelia@file middleware; the UI route has priority: 1 with authelia@file.

Cloudflare returns HTTP 403 / error 1010

Cloudflare error 1010 is generated before the request reaches DefectDojo and means the client signature was blocked. The shared API client sets a stable, browser-compatible User-Agent instead of Python urllib's default signature.

  1. Confirm the workflow uses the current defectdojo/scripts/dd_api.py.
  2. Re-run the failed push workflow.
  3. If the Cloudflare policy requires a specific signature, set DD_USER_AGENT in the workflow environment to that approved value.
  4. Prefer an edge-side exception for the token-authenticated /api/v2/ path if Browser Integrity Check is not useful there; keep normal WAF and rate limiting controls enabled.

Do not disable Cloudflare protection for the UI route to resolve an API-only failure.

Findings not appearing after a CI run

  1. Check the Forgejo workflow Import SecOps reports into DefectDojo step. It runs only on pushes to main or dev. It is continue-on-error: true, so CI passes even when the strict importer marks the upload step as failed.
  2. Re-run the import locally with strict mode: 
    DD_API_URL=... DD_API_TOKEN=... DD_STRICT=1 make defectdojo-import
  3. Verify the token still exists and is a v2 token in DefectDojo.
  4. For a 400 about Product, Engagement, or Test lookup, confirm the API user can create context. Name-based imports send product_type_name=Homelab and auto_create_context=true; alternatively pre-seed the hierarchy with dd_bootstrap.py.

3. Disaster Recovery

  1. Restore PostgreSQL from backup.
  2. Restore the defectdojo and database-related secrets for the affected cluster.
  3. Reconcile the affected overlay (defectdojo/overlays/homelab or defectdojo/overlays/local).
  4. For the local cluster, confirm the node.io/defectdojo=true label still exists on the chosen node before reconciling.
  5. Validate the initializer Job, login, and a basic page load.

4. Scaling and Resource Management

kubectl -n defectdojo top pod
kubectl -n defectdojo describe deploy defectdojo-django

If login POST requests or imports restart uwsgi, increase Django resources or reduce process count in defectdojo/overlays/homelab/values.yaml.

5. Maintenance Procedures

  • Rotate DD_ADMIN_PASSWORD, DD_SECRET_KEY, and DD_CREDENTIAL_AES_256_KEY through the local secret workflow.
  • Revalidate the external URL and CSRF trusted origins after ingress changes.
  • Keep a tested emergency superuser recovery path using the uwsgi container.

6. Rollback Strategy

  • Revert the last known-good DefectDojo overlay and Fleet change in Git.
  • If the failure involved bad bootstrap or migrations on disposable data only, consider restoring the database before redeploying.

7. Post-Incident Actions

  1. Update the README or service page if the bootstrap procedure changed again.
  2. Extend this runbook if a new admin recovery pattern was required.
  3. Record whether the issue came from missing secret material, bad bootstrap order, or runtime resource limits.