pulp3 Runbook¶

Metadata¶

Field	Value
Service	pulp3
Criticality	Tier 1
Owner	Platform / Content supply owner
Namespace	pulp3
Clusters	homelab, ozilab (Ozilab)
Last validated	2026-09-11
Related service page	../services/pulp3.md

Trigger Conditions¶

Pulp API or content routes fail.
Mirror sync or publication jobs fail.
Operator reconciliation is unhealthy.
Signing, storage, or DB-backed state becomes invalid.

1. Health Checks¶

kubectl -n pulp3 get pods,svc,pvc,ingressroute
kubectl -n pulp3 get pulp
kubectl -n pulp3 logs deploy/pulp-operator-controller-manager --tail=200

2. Troubleshooting Workflows¶

Inspect the operator and the Pulp custom resource first:

kubectl -n pulp3 describe pulp
kubectl -n pulp3 get events --sort-by=.lastTimestamp | tail -20
kubectl -n pulp3 get secret

Check signing passphrases, admin credentials, and storage readiness before changing manifests.

Ozilab overlay or Fleet bundle reports missing Secret inputs¶

The Ozilab Kustomization must render without private files. Verify only the required resource names; do not print their data:

kubectl -n pulp3 get secret \
  pulp-admin-password \
  debian-signing-key rpm-signing-key \
  debian-signing-passphrase rpm-signing-passphrase \
  -o name

If any are absent, set kubectl to the target cluster and run:

pulp3/overlays/ozilab/bootstrap-secrets.sh --context <ozilab-context>

Then reapply pulp3/overlays/ozilab or force the Fleet bundle to reconcile. The helper reads local operator-owned inputs, applies stable-name Secrets, and does not print their contents.

Ozilab operator and application ordering¶

The operator CRDs must exist before the Pulp custom resource is applied:

kubectl --context <ozilab-context> apply --server-side -k pulp3/operator
kubectl --context <ozilab-context> wait \
  --for=condition=Established crd/pulps.repo-manager.pulpproject.org \
  --timeout=120s
kubectl --context <ozilab-context> apply -k pulp3/overlays/ozilab

For Fleet, the root Pulp bundle manages only the Ozilab application overlay. The operator remains a cluster prerequisite; do not copy its manifests into the application overlay.

3. Disaster Recovery¶

Reapply pulp3/operator and wait for its CRDs.
Restore the five signing/admin Secrets with the out-of-band helper.
Restore database and content storage if needed.
Reapply the active application overlay.
Wait for the Pulp CR to become healthy.
Validate API, content, and a test sync path.

Restoring the database from a nightly dump (homelab)¶

The pulp-db-dump CronJob writes pulp-<UTC timestamp>.dump (pg_dump custom format) to /mnt/SSD-zpool/apps-data/pulp3/db-dumps on the NAS every night at 23:40. Older dumps are in the dataset's snapshots (.zfs/snapshot/<snap>/pulp3/db-dumps/) and in the TANK/ARCHIVES replicas. The pulp-db-fields-encryption Secret must be the one in place when the dump was taken, or encrypted fields (remote credentials) are unreadable.

# Stop the writers. Scale the operator first, or it scales Pulp back up.
kubectl -n pulp3 scale deploy pulp-operator-controller-manager --replicas=0
kubectl -n pulp3 scale deploy pulp-api pulp-content pulp-worker --replicas=0

# Copy the dump into the database pod and restore it.
scp kardudu@192.168.1.2:/mnt/SSD-zpool/apps-data/pulp3/db-dumps/<file>.dump ./pulp.dump
kubectl -n pulp3 cp ./pulp.dump pulp-database-0:/tmp/pulp.dump
kubectl -n pulp3 exec pulp-database-0 -- sh -c \
  'pg_restore -U "$POSTGRES_USER" --clean --if-exists -d pulp /tmp/pulp.dump && rm /tmp/pulp.dump'

# Restart. The operator restores the API, content and worker replicas.
kubectl -n pulp3 scale deploy pulp-operator-controller-manager --replicas=1

4. Scaling and Resource Management¶

kubectl -n pulp3 top pod

Scale worker or API resources in Git if sync or publication jobs queue up.

5. Maintenance Procedures¶

Rotate admin and signing credentials.
Revalidate exposed URLs after ingress changes.
Schedule sync-heavy changes to avoid resource starvation.
Before forcing or restoring the Ozilab Pulp bundle, verify all five Secrets and the operator CRDs on ozilab.

6. Rollback Strategy¶

Revert the operator or overlay revision.
Restore the previous DB and content snapshot if reconciliation broke runtime state.

7. Post-Incident Actions¶

Add a changelog fragment describing the remediation.
Update the service page if overlays or operational assumptions changed.
Extend this runbook when a new operator failure mode is found.