Skip to content

awx-operator Runbook

Metadata

Field Value
Service awx-operator
Criticality Tier 1
Owner Platform / Automation owner
Namespace awx
Clusters homelab, ozilab
Last validated 2026-08-26
Related service page ../services/awx-operator.md

Trigger Conditions

  • The Fleet bundle <cluster>-awx-operator is not 1/1.
  • awx-web or awx-task is not Ready; the AWX UI is unreachable.
  • Jobs stay queued because the task pod cannot reach Postgres or the projects volume.
  • The AWX custom resource is stuck reconciling.

1. Health Checks

kubectl -n awx get pods,svc,pvc,ingressroute
kubectl -n awx get awx
kubectl -n awx logs deployment/awx-operator-controller-manager -c awx-manager --tail=200
kubectl -n awx logs deployment/awx-web --tail=100

Fleet side (run against the Rancher management cluster):

kubectl -n fleet-default get bundles | grep awx-operator
kubectl get bundledeployments -A | grep awx-operator

A healthy bundle shows 1/1 with an empty status column. ErrApplied with a post render message means the post-render Kustomize layer stopped being self-contained — see section 2.

2. Troubleshooting Workflows

Bundle in ErrApplied with a post-render error

Fleet builds overlays/<cluster>/fleet on an in-memory filesystem with LoadRestrictionsRootOnly, without Helm and without a git binary. Any of these in that directory will fail the bundle:

  • a ../ path traversal ('base' must resolve to a file),
  • a remote git URL (no 'git' program on path),
  • a helmCharts: block.

Reproduce locally — this must succeed with no --enable-helm:

kubectl kustomize awx-operator/overlays/homelab/fleet

Then confirm the full local path still builds:

kubectl kustomize awx-operator/overlays/homelab --enable-helm | head -20

Operator Deployment fails to apply with an immutable selector error

Expected once, when moving a cluster from the old Kustomize base to the chart. spec.selector cannot be changed in place:

kubectl -n awx delete deployment awx-operator-controller-manager

Fleet recreates it on the next reconcile. AWX itself stays up throughout.

AWX CR not reconciling

kubectl -n awx describe awx awx
kubectl -n awx logs deployment/awx-operator-controller-manager -c awx-manager --tail=300 | grep -i "failed\|error"

Check the storage classes named in fleet/awx-deploy.yaml actually exist on the cluster and that the projects PVC is Bound.

3. Disaster Recovery

Infrastructure comes back from Fleet; logical configuration comes back from awx-operator/config/; encrypted secrets and job history come back only from a database backup. Know which of the three you are restoring.

  1. Confirm the CRDs exist (kubectl get crd | grep awx). Fleet installs them from the chart's crds/ directory; the local path needs includeCRDs: true.
  2. Restore the awx-secret-key Secret before anything else — without it, stored credentials cannot be decrypted.
  3. Reconcile the overlay (Fleet re-applies from Git, or kubectl kustomize awx-operator/overlays/homelab --enable-helm | kubectl apply -f -).
  4. Restore Postgres from the latest nightly dump (homelab, see below), or restore Postgres and the projects volume from an AWXBackup PVC using an AWXRestore custom resource.
  5. If Postgres started empty (no restore), rebuild the logical configuration from Git instead — this needs only kubectl and ansible-core, and does not require AWX to already contain a job template:
cd awx-operator/config
ansible-galaxy collection install -r requirements.yml -p ./.collections
export CONTROLLER_HOST=https://awx.home.mutana.fr
export CONTROLLER_OAUTH_TOKEN=$(./scripts/bootstrap-token.sh homelab)
export CONTROLLER_VERIFY_SSL=true
ansible-playbook playbooks/reconcile.yml

Credential secret values are not in Git. Export the AWX_CRED_* / AWX_NOTIFY_* variables listed in the migration report before this run, or the credential objects come back empty. Then revoke the bootstrap token with ./scripts/revoke-token.sh — not with awx-manage revoke_oauth2_tokens --user admin, which revokes every token that user owns, including the one the Windmill AWX resource uses. 6. Verify login, then run one known-good job template end to end.

Restoring the database from a nightly dump (homelab)

The awx-db-dump CronJob writes awx-<UTC timestamp>.dump (pg_dump custom format) to /mnt/SSD-zpool/apps-data/awx/db-dumps on the NAS every night at 23:20. Older dumps are in the dataset's snapshots (.zfs/snapshot/<snap>/awx/db-dumps/) and in the TANK/ARCHIVES replicas.

# Check the last run.
kubectl -n awx get cronjob awx-db-dump
kubectl -n awx logs job/$(kubectl -n awx get jobs -l app.kubernetes.io/name=awx-db-dump \
  --sort-by=.metadata.creationTimestamp -o name | tail -1 | cut -d/ -f2)

# Stop the writers. Scale the operator first, or it scales AWX back up.
kubectl -n awx scale deploy awx-operator-controller-manager --replicas=0
kubectl -n awx scale deploy awx-web awx-task --replicas=0

# Copy the dump into the database pod and restore it.
scp kardudu@192.168.1.2:/mnt/SSD-zpool/apps-data/awx/db-dumps/<file>.dump ./awx.dump
kubectl -n awx cp ./awx.dump awx-postgres-15-0:/tmp/awx.dump
kubectl -n awx exec awx-postgres-15-0 -- \
  pg_restore -U postgres --clean --if-exists --no-owner --role=awx -d awx /tmp/awx.dump
kubectl -n awx exec awx-postgres-15-0 -- rm /tmp/awx.dump

# Restart. The operator restores the web and task replicas.
kubectl -n awx scale deploy awx-operator-controller-manager --replicas=1

The awx-secret-key Secret must be the one that was in place when the dump was taken (step 2).

Checking configuration drift

Read-only, safe at any time:

cd awx-operator/config
ansible-playbook playbooks/verify.yml

Reports declared / missing / drifted / unmanaged. reconcile.yml converges anything it finds; it never deletes.

4. Scaling and Resource Management

kubectl -n awx top pod
kubectl -n awx get pvc

Task and web resource requests, Postgres sizing, and max_connections are set in overlays/<cluster>/fleet/awx-deploy.yaml. Change them in Git; the operator rolls the managed workloads.

5. Maintenance Procedures

  • Chart and operator upgrades: bump the pin in both fleet.yaml and overlays/<cluster>/kustomization.yaml, then run make validate.
  • Take an on-demand backup before any upgrade:
kubectl -n awx apply -f awx-operator/overlays/homelab/fleet/awx-backup.yaml
  • Rotate the AWX admin password through the AWX UI, not by editing Secrets.

6. Rollback Strategy

  • Revert the overlay to the previous working revision and let Fleet reconcile.
  • If the operator version changed, roll the chart pin back in both files; the operator downgrade path is only safe when no AWX schema migration ran.
  • Restore from the most recent nightly dump or AWXBackup if a migration damaged the database.

7. Post-Incident Actions

  1. Add a changelog fragment covering manual recovery.
  2. Update the service page if hostnames, storage classes, or versions changed.
  3. Add the observed failure mode to this runbook.