Skip to content

forgejo-alert-bridge Runbook

Metadata

Field Value
Service forgejo-alert-bridge
Criticality Tier 3
Owner Platform owner
Namespace prometheus
Clusters jls
Last validated 2026-07-04
Related service page ../services/forgejo-alert-bridge.md

Trigger Conditions

  • A known pod or deployment problem has persisted for more than 2 days but no Forgejo issue was filed.
  • An alert resolved but its Forgejo issue stayed open.
  • Alertmanager logs show repeated webhook delivery failures to forgejo-alert-bridge.prometheus.svc.cluster.local:8080.
  • The forgejo-alert-bridge Deployment has no Ready pods.

1. Health Checks

kubectl -n prometheus get deploy forgejo-alert-bridge
kubectl -n prometheus logs deploy/forgejo-alert-bridge --tail=50
kubectl -n prometheus port-forward svc/forgejo-alert-bridge 8080:8080
curl -s localhost:8080/healthz

/healthz returns {"ok":true} with HTTP 200.

2. Troubleshooting Workflows

No issues filed for a known alert:

  1. Confirm the alert is firing and carries ticket="true": kubectl -n prometheus exec statefulset/prometheus-server -- wget -qO- localhost:9090/api/v1/alerts | jq '.data.alerts[] | select(.labels.ticket=="true")'
  2. Confirm Alertmanager loaded the route: kubectl -n prometheus logs statefulset/prometheus-alertmanager --tail=50 | grep -i error
  3. Confirm the bridge received the webhook: kubectl -n prometheus logs deploy/forgejo-alert-bridge --tail=50
  4. Confirm the Forgejo token is valid and has issue scope on kardudu/k8s.

Issue stayed open after resolve:

  1. Confirm Alertmanager has send_resolved: true on the receiver.
  2. Confirm the resolved webhook reached the bridge (bridge logs).
  3. The bridge matches by fingerprint string in the issue body; verify the fingerprint appears in the open issue body.

3. Disaster Recovery

The bridge is stateless. To recover, redeploy from Git. Forgejo issues are independent records and are not affected by a bridge redeploy.

4. Scaling and Resource Management

One replica is sufficient; the bridge is idempotent (repeat firing notifications are no-ops). Resource requests are 20m CPU and 64Mi memory; limits 200m and 128Mi. Do not add replicas without a reason - concurrent webhooks would race on issue creation and produce duplicate issues.

5. Maintenance Procedures

Image update:

  1. Merge a change to forgejo-alert-bridge/app/**; the build workflow pushes a new SHA tag.
  2. Update overlays/jls/kustomization.yaml images.newTag to the new SHA.
  3. Let Fleet reconcile, or apply manually.

Token rotation:

  1. Create a new scoped token at https://git.mutana.fr/user/settings/applications.
  2. Update the local overlays/jls/forgejo-alert-bridge.env and re-apply the overlay, or update the live Secret directly.

6. Rollback Strategy

Revert the offending commit. The bridge is stateless; rollback is a redeploy from Git. Issues already filed remain open and can be closed manually.

7. Post-Incident Actions

  • If duplicate issues were filed due to concurrent webhooks, close the extras manually and confirm only one replica is running.
  • If a storm of alerts filed many issues, consider tightening the alert thresholds or the repeat_interval in the Alertmanager route.
  • Record any token scope or expiry problem and schedule rotation.