feat(k8sbatch): per-reconcile mutation batcher (Phase 1)

[anngdinh]

Summary

Lands pkg/k8sbatch — a generic per-reconcile batcher that coalesces queued mutations against Kubernetes objects so a single reconcile performs at most one GET + one Spec patch + one Status patch per distinct object, instead of N GETs and N PATCHes for N small mutator functions.

• Public API: Batcher, New, Pending, Flush(ctx), generic MutateSpec[T] / MutateStatus[T].
• Semantics: MergeFromWithOptimisticLock + retry.RetryOnConflict per object (mutators re-run against fresh state on 409). Best-effort across objects via errors.Join. Spec-fail-skips-Status (entry stays queued, both retried on next Flush).
• Phase 1 only: No production call sites are migrated. Phases 2 (re-route existing PatchMutateStatus* helpers) and 3 (migrate reconcilers) are explicit follow-ups.

Spec: docs/superpowers/specs/2026-05-09-k8sbatch-design.md
Plan: docs/superpowers/plans/2026-05-09-k8sbatch.md

Test plan

• 13 envtest specs pass (go test -count=1 ./pkg/k8sbatch/...) — covers happy path, mutator-returns-false, queue ordering, NotFound, Spec-only, Spec+Status ordering, Spec-fail-skips-Status, best-effort across objects, mid-reconcile flush, real API-server 409 retry, identity coalescing, and multi-CR-type batching.
• go vet ./pkg/k8sbatch/... clean.
• Reviewer to confirm pinned golangci-lint v1.59.1 typecheck noise on this package is the same pre-existing repo-wide issue (appears on internal/controller/... too), not a regression.

Scope discipline

• Owner-controller Spec writes (Ingress/Service writing to LBC/NSG Spec) are intentionally not migrated; the API supports them via MutateSpec, but call-site migration is deferred.
• The type-specific cmpopts.IgnoreFields(...) filtering used in the old patchMutateStatusObject debug-diff log is dropped (the new package is generic). Acceptable per the spec.

🤖 Generated with Claude Code

[anngdinh]

Using pkg/k8sbatch — quick guide

What it is

A per-reconcile mutation batcher. Instead of every small helper doing its own GET → mutate → PATCH against the same Kubernetes object, you queue mutators on a *Batcher and Flush(ctx) once at the end of the reconcile. Net effect: one GET + at most one Spec patch + one Status patch per distinct object, regardless of how many helpers touch it.

This PR migrates internal/usecase/lbc_uc/status.go (Phase 2 of the design). The 6 statusAdd* helpers used to make ~18 GET+PATCH cycles for a typical LB-with-listeners-and-pools reconcile — now they coalesce into 1 GET + 1 Status PATCH.

API surface

// pkg/k8sbatch
type Batcher struct { /* unexported */ }

func New(c client.Client) *Batcher
func (b *Batcher) Flush(ctx context.Context) error
func (b *Batcher) Pending() int

// Generic queueing — returning false from a mutator skips the patch.
func MutateStatus[T client.Object](b *Batcher, obj T, mutate func(obj T) bool)
func MutateSpec[T client.Object](b *Batcher, obj T, mutate func(obj T) bool)

That's it — five entry points.

Standard usage (Reconcile pattern)

func (uc *lbcUseCase) ensure(ctx context.Context, lbConfig *v1alpha1.LoadBalancerConfig) error {
    task := &defaultModelDeployTask{
        // ... the rest of the task fields ...
        batcher: k8sbatch.New(uc.k8sRepo.Client()),
    }

    deployErr := task.deploy(ctx)              // queues many MutateStatus calls
    flushErr  := task.batcher.Flush(ctx)       // ONE GET + ONE Status PATCH

    // Flush always runs — even on deploy error — so partial status writes
    // still land. This preserves today's progressive-checkpoint behavior.
    if deployErr != nil {
        if flushErr != nil {
            logger.Warnf("flush of batched status updates failed: %v", flushErr)
        }
        return deployErr
    }
    return flushErr
}

Writing a status helper

Before:

func (t *task) statusAddListener(ctx context.Context, listenerId string, port int) error {
    return t.k8sRepo.PatchMutateStatusLoadBalancerConfig(ctx, t.lbConfig, func(ctx context.Context, obj *v1alpha1.LoadBalancerConfig) bool {
        // compare-or-mutate logic
        ...
        return true
    })
}

After:

func (t *task) statusAddListener(_ context.Context, listenerId string, port int) error {
    k8sbatch.MutateStatus(t.batcher, t.lbConfig, func(obj *v1alpha1.LoadBalancerConfig) bool {
        // SAME compare-or-mutate logic — runs at flush time against a fresh GET
        ...
        return true
    })
    return nil
}

The diff is mechanical: change the wrapping call, drop the ctx parameter from the closure, return nil immediately (errors surface from Flush).

Identity coalescing

The queue is keyed by {GVK, namespace, name}. Two mutators on the same object — even passed via different *T pointers — get merged into one entry and run in queue order against a single fresh GET:

b := k8sbatch.New(client)

// All three target the same LBC → one GET, one PATCH at flush
k8sbatch.MutateStatus(b, lbc, setLoadBalancerID)
k8sbatch.MutateStatus(b, lbc, addListener)
k8sbatch.MutateStatus(b, lbc, addPool)

// Different object → separate GET + PATCH (still one Flush)
k8sbatch.MutateStatus(b, nsg, updateAttachedNodes)

b.Flush(ctx) // 2 GETs, 2 Status PATCHes total

Spec writes (owner controllers)

MutateSpec works identically but routes to client.Patch instead of client.Status().Patch. Used by Ingress/Service controllers that own the Spec of LBC/NSG. Spec-then-Status order is preserved when both are queued for the same object. Not migrated to MutateSpec in this PR — service_uc/build_lbc.go and friends already pre-batch Spec writes at the function level, so the win wouldn't be meaningful there.

Semantics you can rely on

• Mutators run against fresh state at flush time (a fresh Get, not the in-memory t.lbConfig you passed). Your compare-on-fresh-state logic still works exactly as before.
• Returning false skips the patch. No-op mutators don't bump the resource version.
• Conflict (409) replays the closure via RetryOnConflict(DefaultBackoff). Mutators must therefore be idempotent compare-with-desired functions — which they already are.
• Best-effort across objects. A failure on one object doesn't prevent patching other objects in the same Flush. Failed entries stay queued; successful ones clear. Aggregated via errors.Join.
• Spec failure skips Status for that object (entry stays queued, both retried on next Flush).
• Mid-reconcile flushes are allowed. If you need a status field persisted before a downstream system reads it, call b.Flush(ctx) mid-flow; subsequent Mutate* calls re-fill the queue.
• Not goroutine-safe. Reconciles are single-goroutine per object key in controller-runtime — that's the only intended caller.

When to use vs. not

Scenario	Use k8sbatch?
Many small functions each updating one or two fields on the same object during a reconcile (e.g., lbc_uc/status.go)	Yes — biggest wins here
Single function builds the whole desired Spec then patches once (e.g., today's service_uc/build_lbc.go)	No — already 1 GET + 1 PATCH; nothing to coalesce
One-shot Create at the start of a reconcile (e.g., CreateLoadBalancerConfig)	No — there's nothing repetitive to batch
Need partial status persistence on error before returning	Yes — call Flush from your error path the same way ensure does

Common pitfalls to avoid

1. Don't expect t.obj.Status.X to reflect a queued mutation. The mutator runs at flush time against a fresh GET, never against your in-memory copy. (This was already the behavior of PatchMutateStatusObject — verified during the migration.)
2. Don't forget to Flush. Without it, queued mutations are discarded. The pattern is New → queue → Flush at every reconcile exit point.
3. Don't pass two different concrete types under the same name/namespace. It can't happen by accident (GVK is part of the key), but if you wrap the API in something exotic, the type-assertion inside the wrapped closure would panic. The struct comments call this out.

Reference implementation

See pkg/k8sbatch/ for the implementation (~170 LOC) and pkg/k8sbatch/batcher_test.go for 13 envtest specs covering every behavior above. The migration in internal/usecase/lbc_uc/status.go is the canonical example.