Skip to content

mutana-heartbeat Runbook

Metadata

Field Value
Service mutana-heartbeat
Criticality Tier 2
Owner Platform owner
Namespace monitoring
Clusters homelab
Last validated 2026-08-28 (manifest/tests; live apply pending)
Related service page ../services/mutana-heartbeat.md

Trigger Conditions

  • https://mutana.xyz/statut/ reports frozen data for more than 45 minutes or a lost heartbeat after six hours.
  • The CronJob has no recent successful Job.
  • Jobs fail fetching Kuma, self-checking Mutana, negotiating FTPS/FTP, changing directory, uploading, or renaming the target.
  • A monitor name/category exposes information that should not be public.

1. Health Checks

kubectl -n monitoring get cronjob mutana-heartbeat
kubectl -n monitoring get jobs -l app.kubernetes.io/name=mutana-heartbeat --sort-by=.status.startTime
kubectl -n monitoring get pods -l app.kubernetes.io/name=mutana-heartbeat
kubectl -n monitoring get events --sort-by='.lastTimestamp' | tail -n 30

Inspect the latest Job logs without printing Secret values:

kubectl -n monitoring logs -l app.kubernetes.io/name=mutana-heartbeat --tail=200
curl -fsS https://mutana.xyz/wp-content/uploads/status.json

Confirm generated_at is recent and service names are intentionally public. Do not mistake an old but valid JSON file for a successful current publisher run.

2. Troubleshooting Workflows

Confirm the required objects exist:

kubectl -n monitoring get secret mutana-heartbeat-secret
kubectl -n monitoring get configmap mutana-heartbeat-script
kubectl -n monitoring describe cronjob mutana-heartbeat

Create one diagnostic run:

JOB="mutana-heartbeat-debug-$(date +%s)"
kubectl -n monitoring create job --from=cronjob/mutana-heartbeat "$JOB"
kubectl -n monitoring logs -l "job-name=$JOB" -f

Common causes:

  • Kuma fetch failure: verify the Secret-defined base URL is reachable from homelab, the slug exists, and its public group contains only approved monitors.
  • Empty service list: the public status-page metadata has no monitor list or the selected public group is empty. Fix Kuma; do not add private monitor discovery to the script.
  • Mutana self-check down: verify https://mutana.xyz independently. A self-check failure may set overall down even when other monitors are up.
  • FTPS warning followed by success: certificate-verified FTPS was unavailable and the documented plain-FTP fallback succeeded. Confirm the configured hostname with LWS; do not disable certificate validation in code.
  • FTP login/cwd failure: validate the scoped account and FTP_DIR out of band. Do not print the Secret or pass the password directly in a shell command.
  • Rename failure: the temporary file may remain, but the previous status.json is intact. Correct permissions/path and rerun; the next upload overwrites the temporary name.
  • Image pull failure: verify the pinned multi-architecture Python digest still exists before changing it.

3. Disaster Recovery

  1. Suspend the CronJob so repeated failures stop:
kubectl -n monitoring patch cronjob mutana-heartbeat --type=merge -p '{"spec":{"suspend":true}}'
  1. Verify/recreate mutana-heartbeat-secret out of band from the required six keys.
  2. Review the Kuma public page for safe display names.
  3. Reapply kubectl apply -k mutana-heartbeat from Git.
  4. Unsuspend, trigger one Job, and verify both its logs and the remote generated_at.

Do not manually edit status.json to pretend the heartbeat is current. The stale/lost presentation is the safe outage state.

4. Scaling and Resource Management

The CronJob is short-lived and concurrencyPolicy: Forbid prevents overlap. Inspect resource use on a manual run:

kubectl -n monitoring top pod -l app.kubernetes.io/name=mutana-heartbeat
kubectl -n monitoring describe job -l app.kubernetes.io/name=mutana-heartbeat

The initial request is 100m CPU and 64Mi memory. Change it only from observed Job use. Do not increase schedule frequency without reviewing LWS rate limits and the site's 45-minute staleness threshold.

5. Maintenance Procedures

  • Review the Kuma public monitor allowlist whenever a monitor is added or renamed.
  • Rotate FTP credentials by updating the existing Secret and running one manual Job.
  • Re-resolve and review the Python multi-architecture digest during an intentional image update.
  • Run python3 -m pytest mutana-heartbeat/test_heartbeat.py -q and render Kustomize before each manifest/script change.
  • Renew related Mutana deployment documentation if the FTP directory or hosting account root changes.

6. Rollback Strategy

  • Fast stabilization: set spec.suspend: true; the site naturally becomes stale/lost.
  • Manifest rollback: revert the mutana-heartbeat/ Git unit and delete the applied Kustomize resources if needed.
  • Secret rollback: restore the prior scoped credentials out of band only if they remain valid; never commit them.
  • There is no data-schema, PVC, database, or application rollback.

7. Post-Incident Actions

  1. Record whether the failure was Kuma, DNS/TLS, FTP auth/path, LWS rate limiting, or the publisher.
  2. Update the public monitor allowlist if a disclosure issue was involved.
  3. Update this runbook and the service page with any corrected operational command or dependency.
  4. Add a changelog fragment for behavior, recovery, or security changes.
  5. Record validation and remaining live-cluster checks in context/progress-tracker.md.