parser: slice midnight-straddling turns instead of dropping them - #857
parser: slice midnight-straddling turns instead of dropping them#857avs-io wants to merge 1 commit into
Conversation
…d of dropping it Both parseProviderSources (Codex/OTel/network providers) and scanProjectDirs (Claude Code, the highest-traffic provider) range-filtered a whole turn by its first call's timestamp. A turn spanning local midnight — common for long-running autonomous sessions — had every call after the boundary silently dropped from the new day: `codeburn today` reported no usage until the turn ended, and daily history attributed the whole turn's cost to the turn-start day. Slice each turn to only its in-range calls instead of dropping it wholesale, and re-anchor `timestamp` to the first surviving call so day-aggregator's turn-anchored day bucketing (which keys off `timestamp`, not per-call timestamps) attributes the retained calls to the day they actually occurred, not the pre-slice turn's original day. - parseProviderSources: turnSlicedToRange slices a CachedTurn's raw calls, applied in both the servedSources and durableSources loops (both had the identical bug). - scanProjectDirs: classifiedTurnSlicedToRange slices a ClassifiedTurn's assistantCalls post-classification. category/subCategory/retries/hasEdits stay exactly as classified from the complete turn (turn-level judgments, not per-call sums) — only assistantCalls and timestamp are trimmed. The PR-linked subagent fold-anchor mechanism (a parent whose own turns are fully out of range still folds an in-range child with zero parent spend) is untouched: it depends on carriedBranch/spawnPrSets built from the FULL pre-slice turn list, and a slice only ever produces a non-empty assistantCalls when the turn genuinely has in-range calls, so it can't turn a real anchor into a spend-bearing turn or vice versa. - Both slicers share a small `callsInRange` predicate; the two composite objects (CachedTurn vs ClassifiedTurn) are built separately since their shapes genuinely differ. Fixes getagentseal#852. Repro, root-cause diagnosis, and the initial fix shape (filter calls inside the turn rather than dropping it, applied to both provider loops) are from the issue reporter (KENSHI601), verified against the current parser and extended to the Claude Code path and the day-aggregator timestamp reconciliation.
dd110d9 to
1643aa9
Compare
iamtoruk
left a comment
There was a problem hiding this comment.
The today fix is genuine, but as it stands this trades a visible bug for a silent one: periods longer than a day lose usage. Requesting changes on a proven conservation violation.
The blocker, reproduced end-to-end through buildDurablePeriod (the shared headline builder behind CLI/TUI/menubar): turn starting 07-27 23:57 with calls at 23:58 ($0.006) and 00:10 ($0.012). Truth for week/month = 2 calls / $0.018.
| period | main | this PR |
|---|---|---|
| today | 0 / $0 (the bug you're fixing) | 1 / $0.012 ✓ |
| week | 2 / $0.018 ✓ | 1 / $0.006 |
| month | 2 / $0.018 ✓ | 1 / $0.006 |
Mechanism: the backfill parse now slices the turn at the backfillStart..yesterdayEnd boundary, so the cached yesterday keeps only pre-midnight calls — but aggregateProjectsIntoDays stays turn-anchored, so in the period parse the whole turn buckets under yesterday and the today bucket gets zero. unionDaysForPeriod = cache (≤ yesterday) + live (today): the post-midnight calls are in neither. Per-call slicing and per-turn day bucketing are incompatible — the "out of scope" note on aggregateProjectsIntoDays is exactly where the gap opens. On a real month of data: 12 straddling turns, $8.98 silently gone.
Second blocker — half-applied across surfaces: filterProjectsByDateRange (dashboard Today/7-Days) and filterProjectsByDays (menubar/history) still whole-turn-drop, so the same data yields different Today numbers per surface — the exact divergence class the durable-totals parity suite exists to prevent (which has no midnight-straddling case; one should land with whichever PR carries this fix).
Also worth stating: turn-level aggregates (editTurns, categories, workflow) now report on BOTH sides of a split — a turn whose only Edit happened entirely on day N reports editTurns=1 on day N+1 too. If that's the intended semantics it needs documenting; summed-across-days turn counts inflate.
Given your own note that #853 should take precedence: agreed — suggest closing this in favor of landing the conservation-correct version once, there. Two things to carry over: (a) #853 currently does NOT re-anchor timestamp after slicing, so the two implementations diverge on the split rule — whichever survives should pin down one rule; (b) #853 doesn't touch scanProjectDirs, so its exposure at the cache-backfill boundary is structurally the same question and needs the same regression test (a straddling-turn buildDurablePeriod case asserting day-N + day-N+1 == whole-range for calls, cost, and tokens — I have a failing test ready to contribute).
Side-finding while auditing: parseSessionFile (~:1832-1846) is dead code carrying a fourth copy of the old filter and the historically-important "orphan assistant calls" comment — worth a separate cleanup so the next person doesn't have to reconcile four filter sites.
Extends the per-call range filter so multi-day periods stop losing usage (review on getagentseal#857): - Re-anchor a sliced turn's timestamp to its first surviving call in parseProviderSources, so every turn slicer shares one split rule. - scanProjectDirs (Claude Code path): slice per call instead of dropping the whole turn on its first assistant timestamp; category/ subCategory/retries/hasEdits stay classified from the full turn. - aggregateProjectsIntoDays: bucket cost/calls/tokens (and the model, project, provider-slice rollups built from them) under each call's own day; turn-level stats (categories, editTurns, oneShotTurns) stay turn-anchored. This is the conservation fix: cache (<= yesterday) + live (today) unions now sum to the whole range for straddling turns. - buildJsonReport's dailyMap fallback follows the same per-call rule so the no-durable path can't diverge from durable.days. - filterProjectsByDateRange (dashboard) and filterProjectsByDays (menubar/history) slice per call instead of dropping whole turns. Adds the straddling-turn buildDurablePeriod case to the durable-totals parity suite (day-N + day-N+1 == whole-range calls/cost/tokens, verified to fail without the fix), covers the today view and the surface filters, and makes the suite hermetic on machines with real provider data.
Relates to #852.
The defect
Both
parseProviderSources(Codex, OTel, network providers) andscanProjectDirs(Claude Code) range-filtered a whole turn by its first call's timestamp. A turn spanning local midnight — common for long autonomous sessions — had every call after the boundary silently dropped from the new day:codeburn todayreported no usage until the turn ended, and daily history attributed the whole turn's cost to the turn-start day.Delta over #853
#853 fixes the two
parseProviderSourcesloops. This branch additionally carries:1. The Claude Code path.
scanProjectDirshas the structurally identical whole-turn drop, filtering onturn.assistantCalls[0].timestamp. Claude Code is the highest-traffic provider in this tool, so the reported symptom persists for most users until this path is fixed too. Handled byclassifiedTurnSlicedToRange, which slicesassistantCallsand leavescategory/subCategory/retries/hasEditsas classified from the complete turn — they are computed once ahead of the date filter, forcarriedBranch/PR-ref tracking reasons already in the code, and are turn-level judgments rather than per-call sums.2. A
timestampre-anchor on a sliced turn.day-aggregator.tsbuckets byturn.timestamprather than per-call timestamps — deliberately, perae18b1f. Without re-anchoring the sliced turn to its first surviving call, the retained post-midnight calls are still filed under the pre-slice day, so they reappear inparseAllSessionsoutput but remain invisible incodeburn today, which is the reported symptom. Verified againstgetDateRange('today')andaggregateProjectsIntoDaysunder a simulated UTC+8 clock.Fold-anchor safety
The PR-linked subagent fold anchor — a parent whose own turns fall entirely out of range is kept as a zero-cost anchor so an in-range child can still attribute — is unchanged.
carriedBranch,carriedPrRefsandspawnPrSetsare built fromcachedFile.turns, the full pre-slice list, before the date filter runs. A slice yields a non-empty call list only when the turn genuinely has an in-range call, so a fully out-of-range turn still filters to nothing. The anchor test passes identically before and after the change, which is the correct obligation: the mechanism was not broken and must not break.Testing
Two regression files built from @KENSHI601's synthetic repro, made deterministic across machine timezones rather than dependent on the runner's local zone:
tests/parser-turn-midnight-range.test.ts(provider path) andtests/parser-claude-turn-midnight-range.test.ts(Claude Code path).Each covers: a turn spanning the boundary attributes post-boundary calls to the new day and pre-boundary calls to the prior day; a same-day control turn is unchanged; a turn entirely outside the range is still excluded; the unfiltered monthly/total aggregate does not double-count; and
aggregateProjectsIntoDayswithgetDateRange('today')reconciles under a simulated UTC+8 clock. The Claude Code file adds the fold-anchor case.All confirmed failing before the change. Full suite 2475 passed,
tsc --noEmitclean.Out of scope
aggregateProjectsIntoDays(dateKey(call.timestamp)instead of the turn day) is deliberately not included. Once turns are sliced per range, a day-scoped query already sees only that day's calls. Reintroducing raw per-call bucketing on a wide unsliced parse — the common path for history and menubar rollups — would resurrect the reconciliation bugae18b1ffixed, where a midnight-straddling turn split across two days andhistory.dailystopped reconciling to the live headline.filterProjectsByDateRange/turnIsInDateRange, the dashboard's in-memory re-narrowing over an already-wide parse, has the same shape but operates on turns already merged intoProjectSummary. Not covered here; a candidate follow-up.Credit
Root-cause diagnosis, the synthetic repro, and the fix shape are @KENSHI601's, in issue #852 and PR #853.