cc-obs-kube Fleet recovery¶

The Jelastic observability cluster is named cc-obs-kube in Rancher. It uses the existing jls overlay directories for Traefik, Authelia, Prometheus, Mimir, Loki, Kubernetes Monitoring, and Garage. The Fleet management context is local; the downstream context is cc-obs-kube.

Verify deployment¶

kubectl --context=local -n fleet-default get gitrepo cc-obs-kube
kubectl --context=local -n fleet-default get clusters.fleet.cattle.io c-p6c5n
kubectl --context=local -n cluster-fleet-default-c-p6c5n-d3cae01819e0 get bundledeployments.fleet.cattle.io
kubectl --context=cc-obs-kube get nodes
kubectl --context=cc-obs-kube get pods,pvc -A

Verify that each application BundleDeployment selects overlays/jls or overlays/jls/fleet and reports a successful deployment. Check application workloads and resource counts as well as the display state: a Helm release containing no workload resources can report Ready when no customization matched.

WaitApplied and DNS failure¶

Inspect the downstream Fleet agent logs and its node before forcing Git sync. Compare DNS from an affected pod with a pod on another node. Inspect the kube-dns service and its EndpointSlices. A Running agent can still fail DNS and never check in with Fleet.

For the 2026-10-20 incident, cordoning node70062, recreating its Weave pod, and recreating the affected Fleet, Rancher, ingress, and node-detector pods restored connectivity. Fleet and the Rancher webhook were relocated while networking was repaired. Fresh-pod DNS checks and replacement ingress and node-detector DNS checks passed before the worker was uncordoned. Do not change the provider-managed kube-proxy or Weave network ranges as part of this recovery.

First Traefik deployment¶

The existing Traefik base references the following upstream CRDs. On a cluster where those CRDs are absent, Helm cannot validate custom resources in the same manifest before the definitions exist. Bootstrap the existing dependency:

kubectl --context=cc-obs-kube apply --server-side --field-manager=cc-obs-fleet-bootstrap --dry-run=server -f https://raw.githubusercontent.com/traefik/traefik/v3.6/docs/content/reference/dynamic-configuration/kubernetes-crd-definition-v1.yml
kubectl --context=cc-obs-kube apply --server-side --field-manager=cc-obs-fleet-bootstrap -f https://raw.githubusercontent.com/traefik/traefik/v3.6/docs/content/reference/dynamic-configuration/kubernetes-crd-definition-v1.yml
kubectl --context=cc-obs-kube wait --for=condition=Established --timeout=45s crd/ingressroutes.traefik.io crd/middlewares.traefik.io

Retry the Traefik bundle after CRD establishment, then follow the remaining bundles. Preserve the existing chart versions, overlays, credentials, storage classes, and service names while diagnosing startup errors.

Monitoring chart hook¶

The jls Kubernetes Monitoring customization sets helm.timeoutSeconds: 600 to allow cold image pulls and the chart's k8s-monitoring-add-finalizer hook to finish. Inspect the hook Job, ServiceAccount, and events when a Helm operation times out. Preserve the chart's hook and RBAC behavior.

External prerequisites¶

The owner approved Garage as the replacement object store: 50 GiB usable, three copies on separate worker-local disks, and fresh buckets. Follow the Garage runbook to prepare retained local storage, bootstrap peers and layout, and create runtime Secrets without exposing their values. The jls Loki/Mimir overlays use the internal Garage endpoint and separate keys; existing application PVCs and former credential generators are retained. Historical IDrive objects were not migrated. Configuration rollback alone cannot copy new data back or repair the former store's denied access.

During cutover, the old crash-looping StatefulSet pods blocked rolling updates. After confirming each StatefulSet template referenced its new Garage Secret, recreating only the affected pod started the updated template and preserved its PVC. Garage's Fleet bundle enables helm.takeOwnership to adopt the resources created for initial bootstrap. All seven application bundles then reported Ready.

The owner confirmed Jelastic should use node ports 80/443. The jls Traefik overlay now uses a host-network worker DaemonSet and an internal ClusterIP Service instead of a Pending LoadBalancer. All three worker /ping checks and HTTPS requests for auth.mutana.site returned 200 after Fleet rollout. Direct worker HTTPS diagnostics bypassed certificate trust solely for the reachability check. The owner then updated Cloudflare DNS to the current cluster public IP. A request through the public Cloudflare endpoint from the cluster returned HTTP 200 with normal TLS verification passing. This public check validates the client-to-Cloudflare certificate; it does not independently audit Cloudflare's origin TLS settings. See the Traefik runbook for verification, certificate-state constraints and rollback.

Source integration¶

An incident patch to a generated Bundle can be replaced when GitRepo syncs a new commit. The root fleet.yaml selectors must therefore be corrected in source. If the GitRepo is temporarily pinned to a recovery branch, keep that branch selected until the reviewed fix is present on main, then restore the GitRepo's main branch and verify all seven effective overlay selections again.