traefik Runbook

Metadata

Field	Value
Service	traefik
Criticality	Tier 0
Owner	Platform / Networking owner
Namespace	traefik
Clusters	homelab, local, jls, ozilab (Ozilab), cc-wrk-kube
Last validated	2026-04-17
Related service page	../services/traefik.md

Trigger Conditions

• External services become unreachable or return TLS errors.
• Dashboard URL stops responding or enters an auth loop.
• LoadBalancer external IP is missing or wrong.
• Let's Encrypt renewals fail or certificates expire unexpectedly.

1. Health Checks

Use these commands first to establish scope.

kubectl -n traefik get deploy,ds,svc,pod,pvc,ingressroute,middleware,tlsoption,tlsstore
kubectl -n traefik get pods -o wide
kubectl -n traefik describe pod -l app.kubernetes.io/app=traefik
kubectl -n traefik logs deploy/traefik --tail=200
kubectl -n traefik get svc traefik -o wide

Probe verification

Traefik uses /ping for readiness and liveness on port 80.

kubectl -n traefik describe pod <traefik-pod-name>
kubectl -n traefik port-forward deploy/traefik 9000:80
curl -I http://127.0.0.1:9000/ping

Record:

• whether the pod is failing readiness, liveness, or both
• whether /ping is healthy but the Service lacks a LoadBalancer IP
• whether a middleware or certificate issue is the actual user-facing problem

cc-obs-kube Jelastic worker ports

The jls overlay runs a host-network DaemonSet on the three untainted workers. Its Service is ClusterIP; Pending external IP and MetalLB checks do not apply to this overlay. The managed API enforces NodePorts 30000-32767, so do not try to allocate NodePorts 80/443 or change provider API settings.

kubectl --context cc-obs-kube -n traefik get ds,pods,svc,pvc -o wide
kubectl --context cc-obs-kube -n traefik rollout status ds/traefik
kubectl --context cc-obs-kube -n traefik logs ds/traefik --tail=100
kubectl --context cc-obs-kube get nodes
# From a host on the worker network:
for worker in 172.16.8.168 172.16.8.174 172.16.8.170; do
  curl --fail --connect-timeout 3 "http://$worker/ping"
done

Validate HTTPS with SNI for auth.mutana.site against each worker and the approved Cloudflare Origin CA trust chain. An isolated curl --insecure test confirms routing and TLS reachability only; it does not validate trust. Then test through the actual public entry point. Direct worker checks alone do not prove provider forwarding or public DNS is correct. On 2026-10-20 the owner corrected Cloudflare DNS to the current cluster public IP; the public auth endpoint then returned 200 with normal client TLS verification. When using Jelastic shared load balancers instead of an environment public IP, the provider also requires the custom domain to be bound to the environment. Do not assume the provider's shared public domain tests a dedicated-IP path.

Check for competing listeners on ports 80/443 before rollout. The DaemonSet uses ClusterFirstWithHostNet DNS. Keep the control-plane taint and validate the existing NFS PVC on each worker. Current routes use the Cloudflare Origin certificate, and acme.json was zero bytes at the 2026-10-20 recovery. Do not activate ACME routes while multiple instances share that file; use a single-writer certificate design first. Do not print the file contents.

For migration, stage and verify the DaemonSet before allowing Fleet to prune the old Deployment. Keep traefik-pvc, configuration and Secret names intact. Rollback by reverting the jls ingress change through the recovery branch: Fleet recreates the previous Deployment and removes the DaemonSet. Verify the Deployment and PVC before changing provider forwarding. That old revision uses a LoadBalancer and again requires an approved external allocation; it does not preserve worker-port exposure.

cc-wrk-kube routes (local -> 46.30.203.187)

A host served from cc-wrk-kube passes through local Traefik, HAProxy, and the worker DaemonSet. Check them in that order:

# local: route, transport, and pinned certificate exist
kubectl --context oci -n traefik get ingressroute,serverstransport,svc,cm cc-wrk-kube cc-wrk-kube-origin-cert
kubectl --context oci -n traefik logs deploy/traefik --tail=200 | grep -i cc-wrk-kube
# cc-wrk-kube: one ready pod per worker, TLS Secret present
kubectl --context cc-wrk-kube -n traefik get ds,pods -o wide
kubectl --context cc-wrk-kube -n traefik get secret cloudflare-origin-cert

• 502 or "certificate signed by unknown authority" on local: cc-wrk-kube is not serving the pinned origin certificate (Secret missing, or TLSStore not applied).
• 404 from cc-wrk-kube: no IngressRoute there for the Host header.
• Request handled by local instead of cc-wrk-kube: a local IngressRoute still matches the host (it outranks the priority-1 catch-all); delete it.
• 421 Misdirected Request: the cc-wrk-kube IngressRoute sets tls.options; remove it.
• Connection refused or reset: HAProxy has no healthy worker backend, or its source ACL does not include the local node that egresses.

2. Troubleshooting Workflows

LoadBalancer has no external IP or traffic never reaches Traefik

kubectl -n traefik get svc traefik -o yaml
kubectl -n metallb-system get ipaddresspools,l2advertisements
kubectl -n metallb-system logs deploy/controller --tail=200
kubectl -n metallb-system logs daemonset/speaker --tail=200

Check:

• Service type is still LoadBalancer in the active overlay
• the requested address is inside the active MetalLB pool
• MetalLB controller and speaker are healthy

Pod restarts or Traefik fails to boot

kubectl -n traefik logs deploy/traefik --previous
kubectl -n traefik describe pod <traefik-pod-name>
kubectl -n traefik get configmap traefik-config -o yaml

Check:

• invalid static or dynamic config rendered into traefik-config
• missing provider secrets such as cloudflare or do-auth-token
• PVC mount issues on traefik-pvc

Ozilab Fleet reports a missing local Secret input

Fleet must not depend permanently on a gitignored generator input. The Ozilab migration uses two reconciliations so Fleet cannot delete the Secrets it previously generated:

kubectl -n traefik get secret cloudflare basic-auth-user \
  -o custom-columns=NAME:.metadata.name,KEEP:.metadata.annotations.helm\\.sh/resource-policy

Check:

• during phase 1, both Secrets exist and show keep before generator inputs are removed from Git
• after phase 2, both Secrets still exist but are maintained out of band
• traefik/overlays/ozilab renders without any local secret file after phase 2
• the Fleet BundleDeployment no longer reports a missing generator source

If either annotation is absent, stop the phase-2 rollout. Restore the phase-1 revision, let Fleet reconcile it, and verify again without printing Secret data.

Cloudflare returns HTTP 525 for one host

525 is a TLS handshake failure between Cloudflare and the origin, so the request never reaches the backend and the app's own logs stay silent. Compare the failing host against a working one on the same edge IP:

for sni in <failing-host> <working-host>; do
  echo "== $sni"
  openssl s_client -connect <edge-ip>:443 -servername "$sni" </dev/null 2>&1 \
    | grep -E "subject=|alert|SSL handshake has read"
done

tlsv1 unrecognized name (alert 112) on the failing SNI only means Traefik has no certificate for that name and is not allowed to fall back to the default one:

kubectl -n <ns> get ingressroute <name> -o jsonpath='{.spec.tls}'

Check:

• the route uses a bare tls: {} on local and cc-wrk-kube. A certResolver or a TLS option with sniStrict: true (modern@file, intermediate@file, old@file) disables the default-certificate fallback, and those two clusters serve every host from the default TLSStore
• if a certResolver is intended, it exists in that cluster's traefik-static-config.yaml and has actually issued the certificate: kubectl -n traefik exec deploy/traefik -- cat /certs/acme.json with "Certificates": null means nothing was ever issued
• the deployed manifest, not just the Kustomize source. Workloads whose fleet.yaml points at an overlays/<cluster>/fleet directory are served from that copy, which can drift from the overlay next to it

Certificate issuance or renewal fails

kubectl -n traefik logs deploy/traefik --tail=400 | grep -i acme
kubectl -n traefik get secret cloudflare do-auth-token cloudflare-origin-cert
kubectl -n traefik get pvc traefik-pvc

Check:

• DNS provider token secret is present and correct
• ACME storage on traefik-pvc is writable
• certResolver names in IngressRoute or TLS config still match the active static configuration
• the route is on a cluster that has resolvers at all: local and cc-wrk-kube declare none and expect a bare tls: {}

Dashboard route fails or redirects endlessly through auth

kubectl -n traefik get ingressroute traefik-dashboard -o yaml
kubectl -n traefik get middleware
kubectl -n auth get pods
kubectl -n traefik logs deploy/traefik --tail=200 | grep -i auth

Check:

• dashboard host matches the overlay you deployed
• Authelia is reachable on the expected internal address
• middleware reference uses the correct provider namespace or suffix for the active overlay

3. Disaster Recovery

Preconditions

• Confirm whether the outage is caused by MetalLB reachability, Traefik config, or provider credential issues.
• Identify the active overlay for the affected cluster.
• Verify the required secret source files and PVC are available.
• For Ozilab, verify the out-of-band cloudflare and basic-auth-user Secrets exist; do not expect Fleet to load plaintext inputs from Git after phase 2.

Stateful workload recovery

Traefik state is small but important because ACME data lives on a PVC.

1. Preserve or snapshot traefik-pvc when possible before destructive actions.
2. Restore the active overlay secrets and config files.
3. Reapply the correct overlay with Kustomize.
4. Restore the PVC if ACME state was lost and certificate continuity matters.
5. Validate /ping, external IP assignment, and dashboard access.

Cluster rebuild dependency order

1. Cluster networking and MetalLB
2. Namespace traefik and required secrets
3. traefik-pvc or replacement certificate strategy
4. Traefik deployment and CRDs
5. Protected routes and dashboard validation

4. Scaling and Resource Management

Preferred path: adjust the overlay manifests and roll through GitOps.

Use these commands to size the problem before changing resources:

kubectl -n traefik top pod
kubectl -n traefik get deploy traefik -o yaml
kubectl -n traefik describe deploy traefik

Record:

• sustained CPU or memory pressure on the ingress pod
• request volume versus access log behavior
• whether scaling above one replica would conflict with the current ReadWriteOnce ACME storage model

Current guidance: do not increase replicas casually while the ACME state is stored on a single PVC.

5. Maintenance Procedures

• Rotate DNS provider tokens.
• Rotate dashboard credentials and origin certificates.
• Review Traefik image and CRD version alignment before upgrades.
• Validate dashboard hostnames and cross-cluster ExternalName routes after overlay changes.

For Ozilab, set kubectl to the target cluster and bootstrap or rotate the two Secrets using the commands in traefik/README.md. Then either let Fleet reconcile traefik/overlays/ozilab from branch dev, or deploy the same overlay manually:

kubectl apply -k traefik/overlays/ozilab

Do not create a Fleet-only Kustomization or copy the workload manifests for the manual path.

For each task, define:

• Preconditions: active overlay identified and backup of provider secrets available
• Impact window: possible certificate or routing disruption during rollout
• Rollback path: revert overlay revision and reapply previous secrets
• Validation steps: /ping, dashboard route, and one representative service route succeed

6. Rollback Strategy

Document the fastest safe rollback path:

• Revert the affected overlay revision.
• Restore the previous secretGenerator source files.
• If ACME state was corrupted, restore the previous traefik-pvc snapshot or delete the bad state and allow re-issuance when acceptable.

7. Post-Incident Actions

After recovery, always:

1. Update CHANGELOG.md if traffic was manually rerouted, certificates were restored, or secrets were rotated outside the normal workflow.
2. Update the Traefik service page if hosts, middleware chains, provider strategy, or metrics wiring changed.
3. Update this runbook if the incident revealed a missing auth, certificate, or MetalLB diagnostic step.
4. Capture any follow-up work to remove legacy prod overlay assumptions.