Skip to content

AWX Configuration-as-Code — migration report

Metadata

Field Value
Date 2026-09-02
Source instance homelab (awx.home.mutana.fr), AWX 24.6.1
Also inspected ozilab (awx.lab.mutana.fr), AWX 24.6.1
Access path kubectl port-forward svc/awx-service — neither hostname resolves off-LAN
Collection pinned awx.awx 24.6.1
Method read-only API export, translated by hand into awx-operator/config/vars/homelab/
Destructive actions none

The import was verified by running playbooks/reconcile.yml --check against the live instance until every object reported ok. Nothing was written to AWX during the migration.

Which instance holds the configuration

ozilab was checked first. It contains only AWX preload data — the Default organization, Demo Project, Demo Inventory, Demo Job Template, Demo Credential, the built-in execution environments and the four built-in cleanup schedules. Nothing on it was authored by an operator, so config/vars/ozilab/ is a set of empty lists with a README explaining why. It reconciles to changed=0 and removes nothing.

All real configuration lives on homelab, and that is what was imported.

Objects imported into source control

Object type Count File
Organizations 1 vars/homelab/organizations.yml
Custom credential types 1 vars/homelab/credential_types.yml
Credentials (metadata) 8 vars/homelab/credentials.yml
Execution environments 1 vars/homelab/execution_environments.yml
Instance groups 1 vars/homelab/instance_groups.yml
Projects 1 vars/homelab/projects.yml
Inventories 1 vars/homelab/inventories.yml
Inventory sources 1 vars/homelab/inventory_sources.yml
Notification templates 1 vars/homelab/notification_templates.yml
Job templates 40 vars/homelab/job_templates.yml
Workflow job templates 1 (1 node) vars/homelab/workflow_job_templates.yml
Schedules 4 vars/homelab/schedules.yml
Teams 0 vars/homelab/teams.yml
RBAC grants 0 vars/homelab/rbac.yml

All 40 job templates were imported field for field, including per-template become_enabled, verbosity, timeout, every ask_*_on_launch prompt, the six-question survey on image_packer_build, its packer instance group, and the Windmill notification wiring on bootstrap_ansible_user.

No AWX numeric id appears anywhere in vars/. Every reference is by name.

Objects deliberately left externally managed

Object Why
controlplane, default instance groups Created and owned by AWX. Reconciling them means fighting the operator over policy fields AWX recalculates.
AWX EE (latest), AWX EE (24.6.1), Control Plane Execution Environment Created by AWX from its DEFAULT_EXECUTION_ENVIRONMENTS / CONTROL_PLANE_EXECUTION_ENVIRONMENT settings. AWX stores them with an empty pull policy, which awx.awx.execution_environment cannot express — its pull option defaults to missing and rejects "", so declaring them would rewrite an AWX-managed field on every run. They reappear by themselves on a rebuilt instance and are safe to reference by name.
30 managed credential types Shipped by AWX. Only the custom Inventory Vault Password File type is declared.
13 hosts, 6 groups in the homelab inventory Synced from inventories/home/hosts.yml in the project repository by an inventory source with overwrite: true. Declaring them in Git would fight every sync. The host list belongs in the project repository.
The admin user Creating AWX local users from Git means putting initial passwords somewhere. Only role grants are modelled; there are none to model yet.
Job history, stdout, activity stream, fact cache Runtime state, not configuration. Database backup territory.

Secrets requiring external injection

None of these values are in Git. Each is the name of an environment variable that tasks/credentials.yml or tasks/notifications.yml reads at reconcile time, and each is sent only when it is set.

AWX object Field Environment variable
ansible (Machine) ssh_key_data AWX_CRED_ANSIBLE_SSH_KEY
ansible (Machine) password AWX_CRED_ANSIBLE_BECOME_PASSWORD
github (Source Control) password AWX_CRED_GITHUB_TOKEN
docker-registry (Container Registry) password AWX_CRED_DOCKER_REGISTRY_PASSWORD
forgejo-registry (Container Registry) password AWX_CRED_FORGEJO_REGISTRY_PASSWORD
vault_pass (Vault) vault_password AWX_CRED_VAULT_PASS
grafana_alloy_vault (Vault) vault_password AWX_CRED_GRAFANA_ALLOY_VAULT_PASSWORD
Inventory Vault Password (custom) vault_password AWX_CRED_INVENTORY_VAULT_PASSWORD
windmill-webhook (notification) Authorization header AWX_NOTIFY_WINDMILL_AUTHORIZATION

None of them are needed for a routine reconcile. All of them are needed to rebuild AWX from empty.

Finding: a live bearer token is stored in clear text

The windmill-webhook notification template holds a Windmill API bearer token in its notification_configuration.headers.Authorization. AWX does not encrypt webhook headers — the value comes back in clear text from GET /api/v2/notification_templates/ to any account that can read it, and it appears in any raw export.

It is not in Git and never will be. Two consequences:

  1. config/.migration/ is gitignored and its files are written 0600, because a raw export contains this token.
  2. Rotate that token if the export directory has ever been shared, or if the AWX admin credential has been shared more widely than the token's blast radius. Then set AWX_NOTIFY_WINDMILL_AUTHORIZATION and reconcile.

The one intentional change (applied 2026-09-02)

homelab_monitoring_alloy_wf has a single node, built in the AWX UI, which therefore carried a random identifier:

1faa9237-09e9-493c-bf3d-dd133ab23dc4

Git addresses workflow nodes by stable logical name, so the node is declared as deploy_alloy. Reconciling with the UUID still in place would add a second node next to it and run the Alloy deployment twice, so tasks/workflows.yml refuses to proceed while an unmanaged identifier exists and prints the fix.

The fix renames the identifier in place — it does not delete and recreate, so the node, its edges, its prompts and its history are preserved. It was run once, on 2026-09-02:

export CONTROLLER_HOST=... CONTROLLER_OAUTH_TOKEN=...
cd awx-operator/config
./scripts/rename-workflow-node.sh \
  homelab_monitoring_alloy_wf \
  1faa9237-09e9-493c-bf3d-dd133ab23dc4 \
  deploy_alloy

Verified afterwards: the workflow still has exactly one node (id 1, now identifier: deploy_alloy, still pointing at obs_grafana_alloy, edges unchanged), and reconcile.yml --check is changed=0 for the workflow too.

This was the only write made to AWX during the migration. The guard in tasks/workflows.yml remains in place for any future node built in the UI: reconciliation stops rather than silently doubling a workflow branch.

What was explicitly not done

  • No AWX object was deleted, and reconciliation has no prune mode.
  • No object was renamed. The workflow node identifier above is the sole exception: a single in-place PATCH of one internal identifier, applied deliberately, reversible by running the same script with the arguments swapped.
  • No credential was rotated. Every stored secret is untouched.
  • No schedule was enabled, disabled or retimed. All four remain enabled with their original DTSTART.
  • No RBAC was changed.
  • No project SCM URL or branch was changed. scm_branch is empty on the live project, meaning "repository default", and was carried verbatim rather than pinned to a guess.
  • No job, project sync or inventory sync was triggered. update_project: false is set explicitly on the project reconciler.
  • The Fleet/Helm/Kustomize deployment was not modified. config/ is a new, separate directory.

Incident during migration: admin tokens revoked

At the end of the migration the bootstrap token was revoked with

awx-manage revoke_oauth2_tokens --user admin

That command revokes every token the user owns, not only the one just issued. On homelab it revoked three tokens: the bootstrap token, plus two pre-existing admin tokens.

Impact. The Windmill scripts under windmill/u/arnaud/awx/ authenticate to AWX with a Bearer token held in a Windmill AWX resource (awx_launch_job.py and friends read awx["token"]). If that token was one of the two revoked, every Windmill → AWX call now fails with HTTP 401 until a new token is issued and stored in the Windmill resource.

Remediation.

cd awx-operator/config
NEW_TOKEN=$(./scripts/bootstrap-token.sh homelab)
# paste $NEW_TOKEN into the Windmill AWX resource, then verify:
#   Windmill -> u/arnaud/awx/awx_connectivity_test

Consider issuing that token from a dedicated AWX user rather than admin, so a future admin-token revocation cannot take the integration down again.

Prevention. scripts/revoke-token.sh was added; it deletes a single token by id via DELETE /api/v2/tokens/<id>/ and leaves every other token alone. The README, the bootstrap script and the runbook now warn against awx-manage revoke_oauth2_tokens --user.

Known limitations

  1. vars/<cluster>/inventories.yml key order is load-bearing. AWX stores inventory variables as a JSON string and awx.awx.inventory compares it as a string, so reordering keys rewrites the field and shows a change on the next run. Add new keys at the end. This is documented in the file itself.
  2. pod_spec_override formatting is load-bearing for the same reason. It is serialised with to_nice_json(indent=2, sort_keys=false), which reproduces AWX's own formatting exactly.
  3. Notification templates are all-or-nothing. awx.awx.notification_template has no update_secrets option and replaces the whole configuration, so a template with declared secret_headers is skipped unless every one of them is supplied. Skipping is reported and leaves the live object untouched.
  4. AWX-authored execution environments cannot be reconciled because their live pull value is "" and the module rejects it. They are referenced but not managed.
  5. Reconciliation currently uses admin rights. A dedicated awx-cac identity with Organization Admin on Default is the intended end state; the bootstrap path necessarily starts as admin because a rebuilt AWX has no other user.
  6. ozilab is not yet under configuration management in any meaningful sense — it has nothing to manage. Promoting it means copying and adapting files from vars/homelab/, not symlinking them.
  7. Encrypted credential values cannot be exported. The API returns $encrypted$. A disaster-recovery rebuild needs them from an external secret store, or an AWXBackup restore with the original awx-secret-key.

Verification performed

cd awx-operator/config
ansible-galaxy collection install -r requirements.yml -p ./.collections
ansible-playbook --syntax-check playbooks/reconcile.yml playbooks/export.yml playbooks/verify.yml
ansible-playbook playbooks/reconcile.yml --check

Result against live homelab AWX, after the node rename:

PLAY RECAP
localhost : ok=21  changed=0  unreachable=0  failed=0  skipped=4

Every object type reports ok. The windmill-webhook notification template is correctly skipped with its reason (its token is not exported in a routine run).

Drift detection agrees:

ansible-playbook playbooks/verify.yml
# declared in Git  : 52
# missing from AWX : 0
# drifted          : 0
# unmanaged in AWX : 0

Lint, independent of AWX:

make awx-config-lint
# yamllint: clean
# ansible-lint: 0 failures, profile 'production' passed
# syntax-check: reconcile.yml, export.yml, verify.yml