Skip to content

awx-operator

Metadata

Field Value
Service awx-operator
Purpose Ansible automation controller (AWX) and the operator that manages it
Criticality Tier 1
Owner Platform / Automation owner
Clusters homelab, ozilab
Namespace awx
Exposure internal (Traefik IngressRoute, per-cluster hostname)
Stateful yes
Backup class on-demand AWXBackup to PVC
RPO / RTO Backup is operator-triggered on demand, 1 to 4 hours to restore
Last reviewed 2026-08-26

1. Service Overview

The AWX Operator runs the AWX control plane — job templates, inventories, credentials, and scheduled Ansible runs — for the homelab and ozilab clusters. The operator reconciles an AWX custom resource into the awx-web, awx-task and Postgres workloads.

Summary

If AWX fails, scheduled and ad-hoc Ansible automation stops. The operator itself is stateless; AWX state lives in Postgres and the projects PVC.

Dependencies

Dependency Type Why it matters
Traefik ingress Exposes awx-service on the per-cluster hostname
Persistent storage storage Postgres data and the projects volume
Fleet GitOps Reconciles the bundle from the awx-operator path
CRDs (awx.ansible.com) api AWX, AWXBackup, AWXRestore, AWXMeshIngress

2. Architecture Diagram

[Fleet GitRepo] --awx-operator path--> [Helm chart awx-operator 2.19.1]
                                              |  post-render kustomize
                                              v
                        [awx-operator-controller-manager]  (namespace awx)
                                              |  reconciles AWX CR
                                              v
                    [awx-web] + [awx-task] + [awx-postgres-15] + PVCs
                                              ^
                     [Traefik IngressRoute] --/  awx-service:80
  • The chart provides only the operator, its RBAC and the CRDs.
  • The AWX instance itself is a plain manifest in the post-render layer (overlays/<cluster>/fleet/awx-deploy.yaml), not a chart value.

3. Deployment Specifications

Item Value
Source path awx-operator/overlays/homelab, awx-operator/overlays/ozilab
Deployment model Dual-mode Helm: Fleet inflates the chart natively then applies overlays//fleet as post-render; local rendering uses kustomize --enable-helm on the same values file
Namespace awx
Workload kind Deployments plus operator-managed Deployments and StatefulSet
Chart or image version awx-operator chart 2.19.1 (appVersion 2.19.1), pinned in fleet.yaml and overlays//kustomization.yaml (keep in sync)
Config files overlays//values.yaml (single source of truth) plus the shared overlays//fleet/ layer

Cluster mapping

Cluster Overlay path Notes
homelab awx-operator/overlays/homelab awx.home.mutana.fr; projects on truenas-nfs-apps, Postgres on local-path
ozilab awx-operator/overlays/ozilab awx.lab.mutana.fr; projects and Postgres both on local-path

4. Configuration Guide

Environment variables

Variable Source Purpose Secret?
WATCH_NAMESPACE downward API (chart) Restricts the operator to the awx namespace no
AWX spec fields overlays//fleet/awx-deploy.yaml Storage classes, Postgres sizing and args, service type no

ConfigMaps

Resource Path Purpose
awx-operator-awx-manager-config chart template Controller-runtime manager config (leader election, probes)
awx-awx-configmap operator-generated AWX runtime settings, created by the operator from the CR

Secrets management

  • Secret names: awx-admin-password, awx-postgres-configuration, awx-secret-key — all generated by the operator in the awx namespace.
  • Source of truth: the operator. None of these are tracked in Git.
  • Rotation trigger: admin credential rotation or a Postgres migration.
  • Recovery note: restoring AWX requires the original awx-secret-key, or saved credentials become undecryptable. Capture it before any rebuild.

5. Access Protocols

Path URL or endpoint Audience Auth TLS terminates at
Internal awx-service.awx.svc:80 Cluster workloads cluster RBAC Service
External (homelab) https://awx.home.mutana.fr Operators AWX login Traefik (LE-prod)
External (ozilab) https://awx.lab.mutana.fr Operators AWX login Traefik (LE-prod)

6. Operations and Observability

  • Primary health indicators: awx-operator-controller-manager Ready, the AWX CR reporting a successful reconcile, and awx-web / awx-task Ready.
  • Dashboards or alerts: operator metrics on awx-operator-controller-manager-metrics-service:8443 behind kube-rbac-proxy.
  • Log locations: awx-manager container for reconcile output; awx-web and awx-task for application behaviour.
  • Known failure modes: reconcile loops from an invalid AWX spec, Postgres PVC pressure, projects PVC unavailable when NFS is down, and Fleet bundle errors when the post-render layer stops being self-contained.

7. Backup and Recovery Notes

  • Backup method: apply the AWXBackup CR in overlays/<cluster>/fleet/awx-backup.yaml. It is intentionally excluded from resources: so Fleet never triggers a backup job on reconcile.
  • Restore prerequisites: the backup PVC, matching operator version, and the original awx-secret-key.
  • Related runbook: ../runbooks/awx-operator.md

8. Release and Change Notes

  • Current deployed app version: awx-operator 2.19.1.
  • Current chart version: awx-operator/awx-operator 2.19.1.
  • Last significant change: restructured to the dual-mode Helm pattern (prometheus shape). The retired base/ pulled a remote git Kustomize URL that Fleet could not resolve, leaving both bundles in ErrApplied.
  • Migration side effects: the operator Deployment must be deleted once per cluster (immutable selector) and the base-era cluster-scoped RBAC is orphaned. See the workload README.
  • Rollback reference: previous overlay revision in Git.