ozilab-platform: migration manifest¶

Status: decided — hosting, DNS, and custody settled 2026-09-22 Date: 2026-09-22 Source of record: homelab repository at commit a52302f

Summary¶

The ozilab cluster is currently driven from a personal homelab repository. It works — nine workloads, delivered by Fleet, path-scoped so unrelated personal services never reach the cluster. The problem is not delivery. It is ownership, auditability, and provenance: the repository mixes work-cluster configuration with personal services, carries plaintext credentials, and lives on personally owned public hosting.

While ozilab is a lab treated as recette, the new repository is under sole custody — one owner, no collaborators, machine-enforced merge gate. That is a deliberate phase, not a permanent shape: the model is designed so that adding a second human is a single amendment (required approvals 0 → 1) rather than a rebuild.

The proposal is to stand up a dedicated ozilab-platform repository, seeded by extraction from the homelab repository, governed by a written constitution with blocking enforcement from the first commit.

The extraction tooling is written and tested. A dry run produces a 205-file repository that renders all ten overlays and passes documentation and changelog validation. What remains is a set of decisions, listed at the end.

Why not the obvious options¶

Option	Why not
Copy the homelab repository	Imports 88 top-level directories, legacy overlay names mid-migration, and — decisively — 12 tracked secret-bearing files under ozilab paths. The goal is a clean standard; this imports the opposite.
Start blank	Discards the assets that are the actual differentiator: the `make check` harness, changelog-fragment discipline, the service/runbook documentation structure, Fleet targeting patterns, and the agent instruction set. Re-deriving these under deadline produces something worse than what it replaced.
Branch from the legacy work repository	That repository is the thing being modernized. Branching from it inherits its shape and couples the new pipeline's lifecycle to the old one's decommission date. It is an input to migrate from, never a base.
Rewrite history with `git filter-repo`	Preserves per-file blame, but this history has close to zero value to a future teammate, and a rewrite adds risk for no payoff. Critically, it does not un-disclose anything — see below.

Extraction keeps the harness and the standards while starting a history that has never contained a credential.

The finding that drives the timeline¶

An audit of tracked files under ozilab paths found twelve secret-bearing files committed to Git, not the two that were previously understood to be outstanding migration debt.

Tier	Credential	Exposure
1	Debian + RPM package-signing PGP private keys and both passphrases	Anyone holding these can sign packages that every ozilab client trusts. This is supply-chain, not just credential, exposure.
2	Cloudflare tunnel token, Cloudflare DNS API token, Traefik basic-auth credential, Tailscale OAuth client	Network ingress to the cluster.
3	AWX bearer token, Grafana Cloud / Loki / Prometheus credentials	Reaches a Grafana Cloud tenant, so blast radius exceeds ozilab.

Eleven of the twelve carry no placeholder marker, i.e. they appear to hold real values. Four were covered by no .gitignore rule at all; one was ignored but tracked anyway.

These must be rotated regardless of which repository option is chosen. Excluding a file from a copy does not remove it from the source history. The new repository does not fix this; it only stops it recurring. Rotation is tracked in ROTATION.md, generated with the new repository and deliberately unchecked.

A second finding, lower severity: all six metallb overlays are silently excluded from make validate in the homelab repository, because overlay discovery matches kustomization.yaml and metallb uses kustomization.yml. The gate has been reporting success over a workload it never rendered. The new repository's discovery covers both.

What gets built¶

Three repositories during transition, two at steady state.

homelab — unchanged. Personal proving ground; new ideas still get tested here first. ozilab paths are removed only after they run from the new repository, one workload at a time.
ozilab-platform — new, seeded by extraction, single root commit, no inherited history. Carries the harness, the standards, and the nine ozilab workloads reduced to base/ + overlays/ozilab/.
The legacy work repository — feature-frozen, hotfixes only. Migrated workload by workload into (2), each as a pull request citing its source path and commit. Tagged and archived at the end, not deleted.

What makes it different from what it replaces¶

Repository choice is only the vehicle. These are set from commit #1, because retrofitting them is precisely what failed before.

A written constitution. Nine articles, each naming the mechanism that enforces it. An article with no enforcing mechanism is an aspiration and does not belong in it.
No secret in Git, enforced. The new repository scans its entire history on every pull request. The homelab repository cannot do this — a full-history scan there fails on known secret debt, so its scan is limited to the commits a pull request adds. Starting clean is what buys the stronger check.
A blocking gate. make check, plus secret, manifest, and image scans, become required status checks, including for administrators. All four targets already exist and are currently optional; optional is how the drift below accumulated.
Documentation enforced by CI. A changed workload without its service doc, runbook, and changelog fragment fails the build.
The same rules for humans and agents. One instruction file, no exemptions for automation. This is a genuinely unusual control to be able to demonstrate.

The drift this is designed to stop¶

In the homelab repository, WORKFLOW.md documents the ozilab Fleet paths as forgejo, metallb, pulp3, traefik. The actual manifest serves longhorn, traefik, pulp3, awx-operator, k8s-monitoring — and the manifest's own comment records that the committed variant "never matched the live object." Documentation drifted from reality inside a single repository, and was still wrong when someone went looking. A dedicated repository with blocking documentation validation is how that stops.

How it is done¶

The extraction kit lives at bootstrap/ozilab-platform/ in the homelab repository. It is tested end to end.

cd bootstrap/ozilab-platform
./extract.sh --dry-run                      # writes nothing
./extract.sh --target ~/Code/ozilab-platform --repo-url <new remote>

It seeds only from git ls-files, never the working tree — the working tree holds untracked real credentials that .gitignore hides from Git but not from a file copy. It refuses to commit if a denied path, private key material, or a credential-named key with a literal value reaches the target.

Verified on a dry run: 205 files, all ten overlays render (including both Helm-inflating ones), documentation and changelog validation pass.

Sequence¶

Rotate Tier 1 credentials — the signing keys. Before anything else.
Decide hosting and DNS (below), then run the extraction with --repo-url.
Replace the CODEOWNERS placeholders; enable branch protection; confirm all five checks run green once so the forge registers their names.
Rotate Tiers 2 and 3; create the replacement Secrets out of band.
Repoint the Fleet GitRepo at the new repository. Verify reconciliation.
Remove ozilab paths from the homelab repository, one workload at a time.
Begin migrating the legacy work repository, one workload per pull request.

Known follow-ups, carried deliberately¶

Six fleet.yaml files still carry targetCustomizations for non-ozilab clusters. Trimming is deliberate work, not a sed — some diff.comparePatches blocks are load-bearing (the Traefik CRD ones especially).
cloudflared has no documentation in the source at all. Stubs are generated so the gate runs from commit #1; they are marked STUB — not yet written and block any future change to that workload until written.
The Fleet bootstrap manifest keeps its fleet/layer7/ path. Renaming it would invent a convention; it is left for a deliberate decision.

Decisions taken¶

Hosting — the Forgejo instance on ozilab. Work-cluster configuration moves off personally owned public hosting and onto infrastructure inside the work perimeter, reachable only over the work VPN. It also dogfoods the platform.

Two consequences are load-bearing:

Bootstrap paradox. forgejo, longhorn, and traefik cannot be delivered from a forge that requires those three workloads to run. They need a seed source outside ozilab and a cold-start path for the case where all three are down together.
Air-gapped authoring. The repository is authored on a machine with no VPN access and pushed from one that has it. Transport is a signed git bundle — a single verifiable file carrying commits and signatures — not a copied working tree, which carries neither. extract.sh --bundle produces it; this has been tested end to end.
DNS — git.lab.mutana.fr for now. Correct while this is a lab in recette. On promotion to preprod or production it moves to a work-owned zone. That is a planned migration with a runbook, not a DNS edit: it changes ROOT_URL, every clone URL, the Fleet GitRepo repo: field, and webhook callbacks.
Custody — sole. One owner, no collaborators, two signing keys (owner and agent). Details below.

The custody model¶

Sole custody is enforced in layers, because any single layer is bypassable. Full detail lives in .forge/forgejo-hardening.md; the shape is:

Layer	Control	Bypassable by
Reachability	Work VPN only	Anyone else on the VPN
Identity	Registration disabled, sign-in required for read	Forge admin
Ownership	Personal repo, one machine collaborator, `FORCE_PRIVATE`	Forge admin
Branch protection	No direct push, merge whitelisted to owner, all checks required	Forge admin
Authorship	Every commit on `main` signed; governance paths require the owner key	Nobody — signatures cannot be forged
Cluster	—	Whoever holds `kubectl`, node root, Longhorn, or the database

Required approvals are 0, deliberately. A sole owner cannot approve their own pull request, and a second self-controlled account is theatre. The status checks are the reviewer; the pull request exists to produce a reviewable artifact and an audit trail, not an approval.

The strongest layer is cryptographic. Access is granted and revoked by whoever administers the forge; a signature is not. The constitution-authorship check requires that any change to CONSTITUTION.md, CODEOWNERS, .forge/, .forgejo/workflows/, Makefile, or scripts/ be signed by the owner key specifically — the agent key is not sufficient. That is the concrete meaning of "sole guarantor": the law of the repository can only be changed by something holding the owner's private key.

What sole custody does not guarantee¶

Stated plainly, because an overstated guarantee is worse than a weaker one.

Cluster admin outranks forge admin. kubectl exec into the Forgejo pod, node root, Longhorn volume access, or the Postgres database each equal full control of this repository. Sole custody of the repository is meaningless without sole custody of ozilab. Who else holds those is not yet enumerated — until it is, the strength of this model is unknown rather than strong.
A forge admin can disable the signing requirement. They cannot forge a signature, so they cannot author as the owner — but they can turn the rule off and merge something unsigned. Signing buys tamper evidence and attribution, not prevention. The detector is an independently verified off-cluster mirror.
The employer owns the substrate — hardware, network, VPN, hypervisor. A guarantee resting on infrastructure someone else controls is a convention.
Bus factor one, by construction. If the owner is unavailable, nothing merges, is amended, or is recovered. Accepted for the lab phase; it must be revisited before anything depends on this in preprod.

Still open¶

Cluster custody — enumerate everyone holding kubectl, node root, Longhorn, hypervisor, or database access to ozilab.
Promotion trigger — what moves this from recette to preprod, and whether sole custody survives an on-call rotation.
Owner key recovery — where the backup lives, and when restoring it was last tested. A key on one laptop is a cluster outage away from making the Constitution unamendable.
Tier 1 rotation owner — who rotates the Pulp package-signing keys, and the window for redistributing new public keys to clients.

Urgent, independent of all the above¶

The running Forgejo instance appears to be open to anonymous read and self-signup, which would undercut every layer above before it is built.

forgejo-config.env sets FORGEJO__REQUIRE_SIGNIN_VIEW with no section segment. The container's environment-to-ini parses FORGEJO__<SECTION>__<KEY>, so that key resolves to section REQUIRE, key SIGNIN_VIEW, and never reaches [service]. Four other keys in the same file have the same defect. DISABLE_REGISTRATION is absent entirely.

If both hold, anyone who reaches git.lab.mutana.fr can browse the instance and create an account. Verify against the running pod — do not infer:

curl -sS -o /dev/null -w '%{http_code}\n' https://git.lab.mutana.fr/explore/repos
# 200 means anonymous read is open; expect a redirect to /user/login.

The corrected, section-qualified configuration is in .forge/forgejo-hardening.md, ready to apply as its own change.

Recommendation¶

Keep it a monorepo. Do not split platform / apps / pipelines yet. Split when a real boundary appears — different approvers, or different release cadence. Fleet path scoping already provides the isolation without the coordination cost.