A rule that fails to program keeps its quota reservation

[ecv]

What needs to happen

The gateway engine reserves per-tenant quota before it programs a rule into the datapath. Three failure paths leave state behind when that programming does not go to plan.

A rule that passes the quota check and then fails to program holds its reservation. The engine does not add it to the active set, so nothing will ever release it. If the operator then deletes the rule, the reservation is held until the process restarts. Each cycle of that shrinks the node's remaining capacity, and the enforcer denies admission long before the datapath is actually full.

Teardown releases quota only after the datapath removal succeeds. A removal that fails leaves the reservation held. That one self-heals on the next pass, because the rule stays in the active set and gets retried.

Shutdown handles teardown failures asymmetrically. The first failure keeps its rule in the active set. Every later failure drops its rule from the set anyway, so the engine reports state gone that the datapath still holds.

There is one more, a layer down. Programming a rule with several virtual addresses writes them one at a time, and the bookkeeping that lets teardown find them is written only after all of them succeed. A failure partway through leaves earlier writes in the table with nothing tracking them. Orphan recovery does sweep those up, so this is bounded, not permanent.

Why this matters

Quota enforcement is live in the gateway binary with real limits, not a stub, so the drift is real once a rule ever fails to program. The symptom shows up far from the cause: rules get refused on a node whose table is mostly empty, and a restart clears it, which is the worst shape a bug can have for on-call.

Why it is not urgent

Every one of these needs a programming failure to trigger, and the whole feature is pre-production. None of them corrupts forwarding state.

Desired outcome

1. Release the reservation on the failure path so a rule that never programmed does not hold capacity.
2. Handle teardown failures the same way regardless of whether an earlier one already failed.
3. Track partial programming so teardown can find what was written, or unwind it on failure rather than relying on orphan recovery.
4. Cover all of it with tests. The current suite exercises the happy paths and quota denial, but no test asserts what quota or the active set look like after a failure.

Found while reviewing #350.

[ecv]

Line references, all on feat/edge-gateway-02-engine.

Reservation held after a failed apply. internal/gateway/engine.go:135-151. applyRuleLocked calls quota.CheckAndReserve, then returns on datapath.ApplyRule error without releasing. Reconcile only adds the key to e.active on success (engine.go:71), so diffRuleKeys never puts it in toRemove and removeRuleLocked never runs. While the rule stays in desired state this is harmless, because NodeQuotaEnforcer.CheckAndReserve is idempotent per key (quota.go:107-144). Delete the rule after a failed apply and the reservation is stranded for the life of the process.

Worth noting it is live: cmd/galactic-gateway/root.go:108 wires NewNodeQuotaEnforcer(DefaultMaxRulesPerTenant, DefaultMaxRuleTableEntries), not the noop.

Release skipped on failed teardown. engine.go:155-161. RemoveRule error returns before quota.Release. Self-heals, since the key stays in e.active and the next pass retries.

Stop's asymmetry. engine.go:113-120:

if err := e.removeRuleLocked(ctx, key); err != nil && firstErr == nil {
    firstErr = fmt.Errorf("stop: remove rule %s: %w", key, err)
    continue
}
delete(e.active, key)

Once firstErr is set, the guard is false for every later failure, so those keys get deleted from e.active despite the teardown having failed.

Partial register. internal/gateway/kerneldatapath.go:110-131. The Register loop returns on first error, and d.ruleKeysByName[rule.Key] = keys is only reached after the loop and the prune. Keys written before the failure are in rule_table with nothing tracking them, so RemoveRule cannot find them. ReconcileOrphans does sweep them, which is what keeps this bounded.

Coverage. engine_test.go has TestEngine_ReconcileQuotaDeniedSkipsApply and TestEngine_ReconcileApplyErrorReportsUnhealthyKeepsGoing, and fakeQuota already records released. No test asserts released after a failed apply, and there is no remove-failure test at all, which is why all four of these are invisible to the suite.

[privateip]

Checked this against main: items 1, 2, and (partly) 4 were already fixed and tested by d426b22 (fix(gateway): stop leaking quota reservations and rule keys) — applyRuleLocked releases quota on a failed apply, removeRuleLocked releases quota even when datapath removal fails, and Stop keeps a key active on any failed teardown regardless of an earlier one. engine_test.go covers all three.

Item 3 (partial-register tracking in KernelDatapath.ApplyRule) was also fixed by that same commit, but shipped without a test — fakeRuleTable had no way to inject a mid-loop failure. Opened #400 to add that coverage: a failOnNthPut knob on the fake table, plus a test that fails the 2nd VIP of a 3-VIP rule's apply and asserts the pre-failure key is tracked and RemoveRule can find and clean it up. Confirmed it's a real regression test by reverting the tracking loop to its pre-fix shape and watching it fail, then restoring.

With #400 merged, all four desired outcomes in this issue are fixed and covered.