feat(dashboard): extract toolReject/toolError friction from CodeBuddy transcripts - #336
feat(dashboard): extract toolReject/toolError friction from CodeBuddy transcripts#336m0Nst3r873 wants to merge 2 commits into
Conversation
… transcripts readCodebuddyIndexOnce hardcoded interrupt/toolReject/toolError to 0 because CodeBuddy's index.json skeleton carries no tool results, so CodeBuddy sessions never scored friction and never triggered the share-learnings hint (discussion Tencent#328). Tool results live in the sibling messages/*.json blobs instead. Add scanCodebuddyBlobs to read them once (before the token-flush retry loop, so IO isn't amplified) and merge two signals into the scan result: - toolReject: an assistant blob whose extra.toolStatus[callId] is cancelled with result.errorMessage containing "User rejected this command" — CodeBuddy's user-rejection-only marker, which excludes system auto-cancels and interrupt residue. - toolError: a tool blob whose tool-result has isError === true. Both are de-duplicated per callId. interrupt stays 0 (CodeBuddy has no on-disk marker for it). Parsing is fully defensive: any single-blob failure is skipped and the function never throws, so the Stop hook is never destabilized. Blob count is capped to bound hook IO. Verified end-to-end against a real rejection session (1 reject / 2 errors) and 20 ordinary sessions (0/0, no false positives). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Stop hook runs dashboardReportHandler (detached background process) and
contributeCheckHandler (foreground) concurrently. The background process
scans the transcript and writes the interventions snapshot into
events.jsonl; the foreground contribute-check reads that snapshot to score
friction. The foreground read routinely wins the race and sees an
un-flushed (empty) friction snapshot, so friction-driven sessions never
score above threshold and the share-learnings hint is silently dropped.
On CodeBuddy the token-flush retry delays the background write further,
making the miss near-deterministic.
Fix: contribute-check now scans the transcript itself (scanTranscriptStop)
for interrupt/toolReject/toolError instead of depending on the
possibly-unwritten Stop event. correction still derives from events
(a stop->prompt_submit pattern, unaffected by the race). Fully backward
compatible: without a transcript path the original events-snapshot path
runs unchanged.
Add a frictionOnly fast path so the foreground caller skips CodeBuddy's
~1.75s token-flush retry loop — friction comes from the one-shot blob scan
and needs no token data, saving foreground hook budget (4.5s cap).
Verified end-to-end against the real CodeBuddy session that first exposed
the race (3f1ac58e): friction now reads {toolReject:1, toolError:2},
enough to cross the hint threshold.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
9002f23 to
ef4f91f
Compare
Update: added a second commit for a race that blocked the hint end-to-endReal-device testing on CodeBuddy surfaced a follow-on bug that the first commit alone didn't fully resolve, so this PR now carries two commits (rebased onto latest main):
The raceAt Stop, Confirmed on a real CodeBuddy session: the Stop event recorded The fixcontribute-check now scans the transcript itself ( A Verification
|
Problem
readCodebuddyIndexOncehardcodedinterrupt/toolReject/toolErrorto0because CodeBuddy'sindex.jsonskeleton carries no tool results. As a result, CodeBuddy sessions never accumulated any friction score and effectively never triggered the share-learnings hint — friction tracking was Claude-Code-only (see discussion #328).Root cause
CodeBuddy stores tool outcomes not in
index.jsonbut in the siblingmessages/*.jsonblobs. The existing scan only read the skeleton, so the friction signals were invisible to the scorer.Change
Add
scanCodebuddyBlobs, invoked once fromscanCodebuddyIndexbefore the token-flush retry loop (blob contents don't change during that window, so IO is not amplified by the 8 retries). It merges two signals into the scan result:toolReject— an assistant blob whoseextra.toolStatus[callId]hasstatus === 'cancelled'andresult.errorMessagecontaining"User rejected this command". This is CodeBuddy's user-rejection-only fixed marker, verified by reverse-engineering the extension's tool state machine: it excludes system auto-cancels (UNFINISHED TOOL/MalformedToolArgs) and interrupt residue, which also land oncancelledbut never carry this string.toolError— a tool blob whosetool-resulthasisError === true. Executed and rejected tools both carryisError === false, so only genuine execution failures count — matching Claude's "is_error=true and not a reject" semantics.Both counts are de-duplicated per
callId.interruptstays0— CodeBuddy has no on-disk marker for it (an interrupted turn's residualcancelledtools carry no user-rejection string, so they are correctly not miscounted as rejects).Nothing downstream changes:
computeSmartScore, the weights, the threshold, and the Claude/Cursor branches are untouched. ThetoolReject/toolErrorfields already existed onTranscriptScanResult— this only fills them in for CodeBuddy.Safety
Parsing is fully defensive:
readdirand every per-blobstat/readFile/JSON.parse/shape access are guarded, any single-blob failure is skipped, and the function never throws — the Stop hook is never destabilized. Blob count is capped (CODEBUDDY_BLOB_MAX_COUNT) to bound hook IO on pathological directories.Test plan
npx tsc --noEmit— passesnpx vitest run src/__tests__/dashboard-collector.test.ts— 70 passed (7 new CodeBuddy cases: reject counting, error counting, isError=false not counted, per-callId dedup, mixed session, missing messages dir, malformed-blob resilience)npm run build— successrmcommand was rejected →{interrupt:0, toolReject:1, toolError:2}(reject precisely captured, signals not cross-counted)0/0(zero false positives)🤖 Generated with Claude Code