pulp3 Runbook

Metadata

Field	Value
Service	pulp3
Criticality	Tier 1
Owner	Platform / Content supply owner
Namespace	pulp3
Clusters	homelab, ozilab (Ozilab)
Last validated	2026-07-28
Related service page	../services/pulp3.md

Trigger Conditions

• Pulp API or content routes fail.
• Mirror sync or publication jobs fail.
• Operator reconciliation is unhealthy.
• Signing, storage, or DB-backed state becomes invalid.

1. Health Checks

kubectl -n pulp3 get pods,svc,pvc,ingressroute
kubectl -n pulp3 get pulp
kubectl -n pulp3 logs deploy/pulp-operator-controller-manager --tail=200

2. Troubleshooting Workflows

Inspect the operator and the Pulp custom resource first:

kubectl -n pulp3 describe pulp
kubectl -n pulp3 get events --sort-by=.lastTimestamp | tail -20
kubectl -n pulp3 get secret

Check signing passphrases, admin credentials, and storage readiness before changing manifests.

Ozilab overlay or Fleet bundle reports missing Secret inputs

The Ozilab Kustomization must render without private files. Verify only the required resource names; do not print their data:

kubectl -n pulp3 get secret \
  pulp-admin-password \
  debian-signing-key rpm-signing-key \
  debian-signing-passphrase rpm-signing-passphrase \
  -o name

If any are absent, set kubectl to the target cluster and run:

pulp3/overlays/ozilab/bootstrap-secrets.sh --context <ozilab-context>

Then reapply pulp3/overlays/ozilab or force the Fleet bundle to reconcile. The helper reads local operator-owned inputs, applies stable-name Secrets, and does not print their contents.

Ozilab operator and application ordering

The operator CRDs must exist before the Pulp custom resource is applied:

kubectl --context <ozilab-context> apply --server-side -k pulp3/operator
kubectl --context <ozilab-context> wait \
  --for=condition=Established crd/pulps.repo-manager.pulpproject.org \
  --timeout=120s
kubectl --context <ozilab-context> apply -k pulp3/overlays/ozilab

For Fleet, the root Pulp bundle manages only the Ozilab application overlay. The operator remains a cluster prerequisite; do not copy its manifests into the application overlay.

3. Disaster Recovery

1. Reapply pulp3/operator and wait for its CRDs.
2. Restore the five signing/admin Secrets with the out-of-band helper.
3. Restore database and content storage if needed.
4. Reapply the active application overlay.
5. Wait for the Pulp CR to become healthy.
6. Validate API, content, and a test sync path.

4. Scaling and Resource Management

kubectl -n pulp3 top pod

Scale worker or API resources in Git if sync or publication jobs queue up.

5. Maintenance Procedures

• Rotate admin and signing credentials.
• Revalidate exposed URLs after ingress changes.
• Schedule sync-heavy changes to avoid resource starvation.
• Before forcing or restoring the Ozilab Pulp bundle, verify all five Secrets and the operator CRDs on ozilab.

6. Rollback Strategy

• Revert the operator or overlay revision.
• Restore the previous DB and content snapshot if reconciliation broke runtime state.

7. Post-Incident Actions

1. Add a changelog fragment describing the remediation.
2. Update the service page if overlays or operational assumptions changed.
3. Extend this runbook when a new operator failure mode is found.