Define cluster SDLC pipeline: flux-local → vCluster → production

[dapperdivers]

Overview

Formalize the software development lifecycle for cluster changes using our existing tools.

Current State

• flux-local (✅ implemented) — Validates Kustomizations + renders Helm on PRs, posts diffs as comments
• vCluster dev (✅ deployed) — Sandbox for runtime testing
• Renovate (✅ running) — Automated dependency updates via PRs
• Lefthook (✅ implemented) — Pre-commit YAML formatting
• CODEOWNERS (✅ implemented) — Path-based review requirements

Proposed Pipeline

Stage 1: Local Development

• lefthook pre-commit hooks catch formatting issues
• task commands for common operations (volsync, reconcile)

Stage 2: Pull Request

• flux-local test validates all Kustomizations render correctly
• flux-local diff shows exactly what changes in the cluster
• CODEOWNERS requires review for critical paths
• New: Add kubeconform schema validation step
• New: Add policy checks (Kyverno dry-run or conftest)

Stage 3: vCluster Testing (for risky changes)

Not every PR needs vCluster — use it for:

• Gateway API migration (new networking stack)
• CRD changes or operator upgrades
• Path-based Flux scoping restructure
• Any change that could break cluster-wide resources

Workflow:

PR created → flux-local validates → reviewer approves →
deploy to vCluster → verify → merge to main → Flux reconciles prod

Stage 4: Production

• Flux reconciles from main automatically
• Alertmanager notifies on failures
• Tuppr handles Talos/K8s upgrades in maintenance windows
• VolSync backs up stateful data

Tasks

• Document the SDLC in repo README or docs/
• Add kubeconform validation to flux-local workflow
• Create task commands for vCluster test deployment
• Define which change types require vCluster testing
• Add branch protection rules requiring flux-local pass
• Create PR template with checklist (tested in vCluster? Y/N)

Key Insight

flux-local is the gatekeeper, vCluster is the proving ground. Most changes only need flux-local. vCluster is for the scary stuff — networking changes, operator upgrades, structural refactors. This keeps velocity high while managing risk.

Related Issues

• Implement path-based Flux scoping for workload isolation #2536 — Path-based Flux scoping
• Deploy dev vCluster for feature branch testing #2537 — Dev vCluster (✅ deployed)
• Migrate from nginx Ingress to Gateway API (Envoy Gateway) #2539 — Gateway API migration (first vCluster test candidate)
• Add flux-local CI validation on pull requests #2557 — flux-local CI (✅ already implemented)
• Add Kubernetes CRD schema generation workflow #2558 — CRD schema generation (enables kubeconform)