Skip to content

feat(act): measure realized savings for defer-* actions in act report - #862

Open
AVSRPA1KR wants to merge 1 commit into
getagentseal:mainfrom
AVSRPA1KR:feat/defer-report-baselines
Open

feat(act): measure realized savings for defer-* actions in act report#862
AVSRPA1KR wants to merge 1 commit into
getagentseal:mainfrom
AVSRPA1KR:feat/defer-report-baselines

Conversation

@AVSRPA1KR

Copy link
Copy Markdown
Contributor

Follow-up to #631 (part 2 of the deferral-coverage work): make the defer-* actions measurable in act report.

The gap

The defer-enable / defer-alwaysload / defer-threshold plan kinds from #631 apply, journal, and undo correctly, but act report couldn't measure them — it returned "not measurable: no baseline captured at apply time", because report.ts never captured a baseline for these kinds nor knew how to compute their realized delta. The measurement piece the design called for was left unwired.

The fix

Deferral has the same effect as mcp-remove — MCP tool-def schema leaves the upfront prefix — so this mirrors that path, with one inversion in the realized signal:

  • needsConfigBaseline + captureBaseline now cover the defer-* kinds. Servers are the named set for defer-alwaysload, or the observed MCP surface for defer-enable / defer-threshold (which re-enable deferral across everything). Per-session tokens use observed tool counts, or the 5-tools × 400 fallback, exactly like mcp-remove.
  • A new deferRow computes realized savings as per-session prefix tokens × the post-apply sessions where deferral actually became active — detected by the same deferred-tools-inventory signal the mcp-deferral-off detector uses. ENABLE_TOOL_SEARCH is read at process start, so sessions begun before the client restarted still run deferral-off and are excluded; if none benefited, the row reports "not yet in effect" with zero realized, rather than claiming a saving that hasn't taken hold.

Records applied before this change (no baseline) keep the existing "no baseline captured at apply time" note, so nothing regresses.

Tests

11 new tests cover measured, partial, not-yet-in-effect, no-sessions, empty-baseline, and missing-baseline paths, plus baseline capture for each kind. Verified end to end locally: applying a defer-enable captures the baseline, and act report reports realized savings scaled to the sessions that adopted deferral (and honestly reports "not yet in effect" when none have).

Diff is limited to src/act/report.ts (+80) and tests/act-report.test.ts (+140). tsc --noEmit clean; the full act suite passes.

Note on CI: tests/cli-durable-totals.test.ts currently fails on main itself (reproduced on a clean checkout of 146037b) — it's in the daily-cache path, unrelated to this change, which only touches src/act/report.ts.

The defer-enable / defer-alwaysload / defer-threshold plan kinds (part 2
of the deferral-coverage work) applied and undid correctly but were
invisible to `act report` — it returned "not measurable: no baseline
captured at apply time", because report.ts never captured a baseline for
them or knew how to compute their realized delta.

Wire them in, mirroring the mcp-remove path since deferral has the same
effect (MCP tool-def schema leaves the upfront prefix):

- needsConfigBaseline + captureBaseline now cover the defer-* kinds.
  Servers are the named set for defer-alwaysload, or the observed MCP
  surface for defer-enable / defer-threshold (which re-enable deferral
  across everything). Per-session tokens use observed tool counts, or the
  5-tools x 400 fallback, exactly like mcp-remove.
- A new deferRow computes realized savings as per-session prefix tokens
  times the post-apply sessions where deferral actually became active —
  detected by the same deferred-tools-inventory signal the
  mcp-deferral-off detector uses. Sessions begun before the client
  restarted still run deferral-off and are excluded; if none benefited,
  the row reports "not yet in effect" with zero realized rather than
  claiming a saving that has not taken hold.

Records applied before this change (no baseline) keep the existing
"no baseline captured at apply time" note, so nothing regresses.

11 tests cover the measured, partial, not-yet-in-effect, no-sessions,
empty-baseline, and missing-baseline paths, plus baseline capture for
each kind. Verified live end to end: apply captures the baseline, and
act report reports realized savings scaled to the sessions that adopted
deferral.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant