Skip to content

longhorn Runbook

Metadata

Field Value
Service longhorn
Criticality Tier 0 (cluster default storage)
Owner Platform owner
Namespace longhorn-system
Clusters ozilab
Last validated 2026-09-04
Related service page ../services/longhorn.md

Trigger Conditions

  • PVCs stay Pending or pods report volume attach/mount errors.
  • volumes.longhorn.io shows robustness: degraded or faulted.
  • Longhorn UI unreachable at https://longhorn.lab.mutana.fr (401 without credentials is the healthy auth gate).
  • Node reports kubelet disk pressure or a storage node drops NotReady.

1. Health Checks

kubectl --context ozilab -n longhorn-system get pods | grep -vE 'Running|Completed'
kubectl --context ozilab -n longhorn-system get nodes.longhorn.io        # 3 nodes, Ready, Schedulable
kubectl --context ozilab -n longhorn-system get volumes.longhorn.io      # robustness: healthy
kubectl --context ozilab -n longhorn-system get replicas.longhorn.io -o wide
kubectl --context ozilab get sc                                           # exactly one default: longhorn

Expect Longhorn components on the three storage workers only (ozisrrepo01, ozisrparlabtools01, ozisrparlabtools02) — never on ozisrpartrke01 (edge taint; upstream ships no tolerations).

2. Troubleshooting Workflows

PVC Pending / Multi-Attach errors. A claim mounted by several Deployments (awx-projects-claim is used by awx-web and awx-task) must be RWX. RWO pins all consumers to one node; a second pod elsewhere hangs with Multi-Attach error. Recreate such claims with accessModes: ["ReadWriteMany"] (scale consumers to 0 first; Longhorn serves RWX via a share-manager-* pod).

Node stuck NotReady in longhorn. Check the Kubernetes node conditions first — the Longhorn node mirrors them (a 2026-09-04 rollout hiccup was kubelet disk pressure on ozisrrepo01, not a Longhorn fault). Check the disk map matches reality: kubectl -n longhorn-system get nodes.longhorn.io <node> -o jsonpath='{.spec.disks}' — every storage node must point at /mnt/kube-data, which must be a real mount (df -h /mnt/kube-data), not a directory on the 5G root LV.

Disk mismatch warnings (mismatching disks in node resource object and monitor collected data) mean the node CR disk path disagrees with where the marker file lives. With no volumes on the node yet, simply patch spec.disks to the path that actually holds longhorn-disk.cfg.

Kubelet disk pressure on ozisrrepo01. /var (containerd imagefs) is the tight filesystem on the old system-LVM nodes. It was extended to 20G after AWX image pulls triggered evictions twice. If it recurs: lvextend -r -L +5G /dev/vg_system/lv_var (vg_system had ~6G free after the first extension).

3. Disaster Recovery

  • Volumes default to 3 replicas across the 3 storage nodes: one node lost leaves every volume writable on 2 replicas; rebuilds start when the node returns.
  • No backup target is configured on ozilab yet (the homelab overlay's minio backup secret is not deployed here). Setting one up is the highest -value follow-up: Longhorn recurring snapshots to S3/minio.
  • Recreating the longhorn StorageClass or deleting CRDs does not delete replica data on /mnt/kube-data, but uninstalling via the upstream cleanup steps does — never run them casually.

4. Scaling and Resource Management

Adding a storage node (see "Node prep" under Maintenance) automatically registers it with a default disk at /mnt/kube-data (from the longhorn-default-setting CM). Reserve capacity headroom: replica count 3 means every volume consumes space on all three nodes.

Existing local-path PVs (pulp3 500G file storage, awx postgres, alloy WAL, traefik) are NOT on Longhorn; migrate them one at a time only with a maintenance window (scale consumer to 0, copy data to a new Longhorn PVC, repoint).

5. Maintenance Procedures

Node prep for a new storage node (run as root on the node; the initial rollout ran it through a one-shot privileged DaemonSet using nsenter -t 1 -m -u -i -n):

set -euo pipefail
export DEBIAN_FRONTEND=noninteractive
dpkg -s nfs-common >/dev/null 2>&1 || { apt-get update -qq; apt-get install -y -qq -o Dpkg::Options::=--force-confold nfs-common; }
dpkg -s open-iscsi >/dev/null 2>&1 || apt-get install -y -qq open-iscsi
systemctl enable --now open-iscsi >/dev/null 2>&1 || true
systemctl is-active --quiet iscsid || systemctl start iscsid
LV=$(lvs --noheadings -o lv_path -S 'vg_name=data && lv_name=kube' 2>/dev/null | xargs || true)
if [ -n "${LV:-}" ] && [ -b "$LV" ]; then
  blkid "$LV" >/dev/null 2>&1 || mkfs.ext4 -F -L kube-data "$LV"
  grep -q " /mnt/kube-data " /etc/fstab || echo "UUID=$(blkid -s UUID -o value "$LV") /mnt/kube-data ext4 defaults,nofail 0 2" >> /etc/fstab
  mkdir -p /mnt/kube-data
  mountpoint -q /mnt/kube-data || mount /mnt/kube-data
fi

The labtools data/kube LVs were already formatted at build time (fstab UUID entries existed) — the blkid guard correctly skipped mkfs. Do not trust lsblk FSTYPE inside a chroot (stale udev data); always blkid.

Storage node drain.

kubectl -n longhorn-system patch nodes.longhorn.io <node> --type=merge -p '{"spec":{"allowScheduling":false}}'
kubectl -n longhorn-system patch nodes.longhorn.io <node> --type=merge \
  -p '{"spec":{"disks":{"<disk-key>":{"evictionRequested":true}}}}'
# wait for replicas to rebuild elsewhere, then drain/reboot the node

Default StorageClass caveat. k3s re-applies its embedded local-path addon on restarts and may restore the default annotation. Fix: kubectl --context ozilab annotate sc local-path storageclass.kubernetes.io/is-default-class-. Durable fix at the next planned k3s restart: add disable-local-path-provisioner: true to /etc/rancher/k3s/config.yaml on ozisrpartrke01 (no config file exists yet; k3s runs with CLI flags).

6. Rollback Strategy

Longhorn was deployed via kubectl apply -k longhorn/overlays/ozilab (WORKFLOW mode 1). Rolling back a configuration change = fix the overlay, re-apply. Removing Longhorn entirely follows the upstream uninstall procedure (scale deployments down, delete CRDs, remove /mnt/kube-data contents) — treat as destructive: every Longhorn PVC (including awx-projects-claim) must be evacuated first.

If Fleet reconciliation of the longhorn path misbehaves, remove the path from the ozilab GitRepo; the manual apply path keeps working unchanged.

7. Post-Incident Actions

  • Record the incident in CHANGELOG.md and, if operational behavior changed, update this runbook and docs/services/longhorn.md.
  • After any node loss, confirm volumes returned to healthy and replicas rebuilt (get volumes.longhorn.io, get replicas.longhorn.io -o wide).
  • After k3s upgrades on ozisrpartrke01, re-check kubectl get sc for a re-defaulted local-path class.