Skip to content

awx-operator

Metadata

Field Value
Service awx-operator
Purpose Ansible automation controller (AWX) and the operator that manages it
Criticality Tier 1
Owner Platform / Automation owner
Clusters homelab, ozilab
Namespace awx
Exposure internal (Traefik IngressRoute, per-cluster hostname)
Stateful yes
Backup class nightly pg_dump to the NAS (homelab); on-demand AWXBackup to PVC
RPO / RTO 24 hours on homelab (nightly dump), on demand elsewhere; 1 to 4 hours to restore
Last reviewed 2026-09-11

1. Service Overview

The AWX Operator runs the AWX control plane — job templates, inventories, credentials, and scheduled Ansible runs — for the homelab and ozilab clusters. The operator reconciles an AWX custom resource into the awx-web, awx-task and Postgres workloads.

Summary

If AWX fails, scheduled and ad-hoc Ansible automation stops. The operator itself is stateless; AWX state lives in Postgres and the projects PVC.

Dependencies

Dependency Type Why it matters
Traefik ingress Exposes awx-service on the per-cluster hostname
Persistent storage storage Postgres data and the projects volume
Fleet GitOps Reconciles the bundle from the awx-operator path
CRDs (awx.ansible.com) api AWX, AWXBackup, AWXRestore, AWXMeshIngress

2. Architecture Diagram

[Fleet GitRepo] --awx-operator path--> [Helm chart awx-operator 2.19.1]
                                              |  post-render kustomize
                                              v
                        [awx-operator-controller-manager]  (namespace awx)
                                              |  reconciles AWX CR
                                              v
                    [awx-web] + [awx-task] + [awx-postgres-15] + PVCs
                                              ^
                     [Traefik IngressRoute] --/  awx-service:80
  • The chart provides only the operator, its RBAC and the CRDs.
  • The AWX instance itself is a plain manifest in the post-render layer (overlays/<cluster>/fleet/awx-deploy.yaml), not a chart value.
  • AWX logical configuration is not deployed by Fleet at all. Organizations, projects, inventories, credentials, job templates, workflows, schedules and RBAC are reconciled from Git through the AWX API by Ansible, in awx-operator/config/. See Configuration-as-Code and the migration report.
Fleet    -> AWX infrastructure (namespace, operator, AWX CR, Postgres, ingress)
Ansible  -> AWX configuration  (everything behind /api/v2/)
Git      -> source of truth for both, in two separate directories

3. Deployment Specifications

Item Value
Source path awx-operator/overlays/homelab, awx-operator/overlays/ozilab
Deployment model Dual-mode Helm: Fleet inflates the chart natively then applies overlays//fleet as post-render; local rendering uses kustomize --enable-helm on the same values file
Namespace awx
Workload kind Deployments plus operator-managed Deployments and StatefulSet
Chart or image version awx-operator chart 2.19.1 , pinned in fleet.yaml and overlays//kustomization.yaml (keep in sync)
Config files overlays//values.yaml (single source of truth) plus the shared overlays//fleet/ layer

Cluster mapping

Cluster Overlay path Notes
homelab awx-operator/overlays/homelab awx.home.mutana.fr; projects on truenas-nfs-apps, Postgres on local-path
ozilab awx-operator/overlays/ozilab awx.lab.mutana.fr; projects and Postgres both on local-path

4. Configuration Guide

Environment variables

Variable Source Purpose Secret?
WATCH_NAMESPACE downward API (chart) Restricts the operator to the awx namespace no
AWX spec fields overlays//fleet/awx-deploy.yaml Storage classes, Postgres sizing and args, service type no

ConfigMaps

Resource Path Purpose
awx-operator-awx-manager-config chart template Controller-runtime manager config (leader election, probes)
awx-awx-configmap operator-generated AWX runtime settings, created by the operator from the CR

Secrets management

  • Secret names: awx-admin-password, awx-postgres-configuration, awx-secret-key — all generated by the operator in the awx namespace.
  • Source of truth: the operator. None of these are tracked in Git.
  • Rotation trigger: admin credential rotation or a Postgres migration.
  • Recovery note: restoring AWX requires the original awx-secret-key, or saved credentials become undecryptable. Capture it before any rebuild.
  • AWX credential metadata (which credentials exist, of which type, for which organization) is in Git under awx-operator/config/vars/<cluster>/. The secret values are not, and are not recoverable from an API export — the API returns $encrypted$. They come from an external secret source at reconcile time, or from a database restore with the original awx-secret-key. Configuration-as-Code and database backup are complementary, not alternatives.

5. Access Protocols

Path URL or endpoint Audience Auth TLS terminates at
Internal awx-service.awx.svc:80 Cluster workloads cluster RBAC Service
External (homelab) https://awx.home.mutana.fr Operators AWX login Traefik (LE-prod)
External (ozilab) https://awx.lab.mutana.fr Operators AWX login Traefik (LE-prod)

6. Operations and Observability

  • Primary health indicators: awx-operator-controller-manager Ready, the AWX CR reporting a successful reconcile, and awx-web / awx-task Ready.
  • Dashboards or alerts: operator metrics on awx-operator-controller-manager-metrics-service:8443 behind kube-rbac-proxy.
  • Log locations: awx-manager container for reconcile output; awx-web and awx-task for application behaviour.
  • Known failure modes: reconcile loops from an invalid AWX spec, Postgres PVC pressure, projects PVC unavailable when NFS is down, and Fleet bundle errors when the post-render layer stops being self-contained.

7. Backup and Recovery Notes

  • Nightly dump (homelab only): the awx-db-dump CronJob runs pg_dump at 23:20 Europe/Paris into the db-dumps PVC (truenas-nfs-apps), which is /mnt/SSD-zpool/apps-data/awx/db-dumps on the NAS. It keeps 7 days of dumps in the folder; NAS snapshots, the TANK/ARCHIVES replicas and the offsite egress keep older ones. Postgres itself stays on local-path.
  • On-demand backup: apply the AWXBackup CR in overlays/<cluster>/fleet/awx-backup.yaml. It is intentionally excluded from resources: so Fleet never triggers a backup job on reconcile. On homelab its backup_storage_class is truenas-nfs-apps.
  • Restore prerequisites: the dump file or backup PVC, matching operator version, and the original awx-secret-key.
  • Related runbook: ../runbooks/awx-operator.md

8. Release and Change Notes

  • Current deployed app version: awx-operator 2.19.1.
  • Current chart version: awx-operator/awx-operator 2.19.1.
  • Last significant change: restructured to the dual-mode Helm pattern (prometheus shape). The retired base/ pulled a remote git Kustomize URL that Fleet could not resolve, leaving both bundles in ErrApplied.
  • Migration side effects: the operator Deployment must be deleted once per cluster (immutable selector) and the base-era cluster-scoped RBAC is orphaned. See the workload README.
  • Rollback reference: previous overlay revision in Git.