Skip to content

awx-operator Runbook

Metadata

Field Value
Service awx-operator
Criticality Tier 1
Owner Platform / Automation owner
Namespace awx
Clusters homelab, ozilab
Last validated 2026-08-26
Related service page ../services/awx-operator.md

Trigger Conditions

  • The Fleet bundle <cluster>-awx-operator is not 1/1.
  • awx-web or awx-task is not Ready; the AWX UI is unreachable.
  • Jobs stay queued because the task pod cannot reach Postgres or the projects volume.
  • The AWX custom resource is stuck reconciling.

1. Health Checks

kubectl -n awx get pods,svc,pvc,ingressroute
kubectl -n awx get awx
kubectl -n awx logs deployment/awx-operator-controller-manager -c awx-manager --tail=200
kubectl -n awx logs deployment/awx-web --tail=100

Fleet side (run against the Rancher management cluster):

kubectl -n fleet-default get bundles | grep awx-operator
kubectl get bundledeployments -A | grep awx-operator

A healthy bundle shows 1/1 with an empty status column. ErrApplied with a post render message means the post-render Kustomize layer stopped being self-contained — see section 2.

2. Troubleshooting Workflows

Bundle in ErrApplied with a post-render error

Fleet builds overlays/<cluster>/fleet on an in-memory filesystem with LoadRestrictionsRootOnly, without Helm and without a git binary. Any of these in that directory will fail the bundle:

  • a ../ path traversal ('base' must resolve to a file),
  • a remote git URL (no 'git' program on path),
  • a helmCharts: block.

Reproduce locally — this must succeed with no --enable-helm:

kubectl kustomize awx-operator/overlays/homelab/fleet

Then confirm the full local path still builds:

kubectl kustomize awx-operator/overlays/homelab --enable-helm | head -20

Operator Deployment fails to apply with an immutable selector error

Expected once, when moving a cluster from the old Kustomize base to the chart. spec.selector cannot be changed in place:

kubectl -n awx delete deployment awx-operator-controller-manager

Fleet recreates it on the next reconcile. AWX itself stays up throughout.

AWX CR not reconciling

kubectl -n awx describe awx awx
kubectl -n awx logs deployment/awx-operator-controller-manager -c awx-manager --tail=300 | grep -i "failed\|error"

Check the storage classes named in fleet/awx-deploy.yaml actually exist on the cluster and that the projects PVC is Bound.

3. Disaster Recovery

  1. Confirm the CRDs exist (kubectl get crd | grep awx). Fleet installs them from the chart's crds/ directory; the local path needs includeCRDs: true.
  2. Restore the awx-secret-key Secret before anything else — without it, stored credentials cannot be decrypted.
  3. Reconcile the overlay (Fleet re-applies from Git, or kubectl kustomize awx-operator/overlays/homelab --enable-helm | kubectl apply -f -).
  4. Restore Postgres and the projects volume from the AWXBackup PVC using an AWXRestore custom resource.
  5. Verify login, then run one known-good job template end to end.

4. Scaling and Resource Management

kubectl -n awx top pod
kubectl -n awx get pvc

Task and web resource requests, Postgres sizing, and max_connections are set in overlays/<cluster>/fleet/awx-deploy.yaml. Change them in Git; the operator rolls the managed workloads.

5. Maintenance Procedures

  • Chart and operator upgrades: bump the pin in both fleet.yaml and overlays/<cluster>/kustomization.yaml, then run make validate.
  • Take an on-demand backup before any upgrade:
kubectl -n awx apply -f awx-operator/overlays/homelab/fleet/awx-backup.yaml
  • Rotate the AWX admin password through the AWX UI, not by editing Secrets.

6. Rollback Strategy

  • Revert the overlay to the previous working revision and let Fleet reconcile.
  • If the operator version changed, roll the chart pin back in both files; the operator downgrade path is only safe when no AWX schema migration ran.
  • Restore from the most recent AWXBackup if a migration damaged the database.

7. Post-Incident Actions

  1. Add a changelog fragment covering manual recovery.
  2. Update the service page if hostnames, storage classes, or versions changed.
  3. Add the observed failure mode to this runbook.