Skip to content

feat(dashboard): extract toolReject/toolError friction from CodeBuddy transcripts - #336

Open
m0Nst3r873 wants to merge 2 commits into
Tencent:mainfrom
m0Nst3r873:feature/codebuddy-friction-signals
Open

feat(dashboard): extract toolReject/toolError friction from CodeBuddy transcripts#336
m0Nst3r873 wants to merge 2 commits into
Tencent:mainfrom
m0Nst3r873:feature/codebuddy-friction-signals

Conversation

@m0Nst3r873

Copy link
Copy Markdown
Collaborator

Problem

readCodebuddyIndexOnce hardcoded interrupt / toolReject / toolError to 0 because CodeBuddy's index.json skeleton carries no tool results. As a result, CodeBuddy sessions never accumulated any friction score and effectively never triggered the share-learnings hint — friction tracking was Claude-Code-only (see discussion #328).

Root cause

CodeBuddy stores tool outcomes not in index.json but in the sibling messages/*.json blobs. The existing scan only read the skeleton, so the friction signals were invisible to the scorer.

Change

Add scanCodebuddyBlobs, invoked once from scanCodebuddyIndex before the token-flush retry loop (blob contents don't change during that window, so IO is not amplified by the 8 retries). It merges two signals into the scan result:

  • toolReject — an assistant blob whose extra.toolStatus[callId] has status === 'cancelled' and result.errorMessage containing "User rejected this command". This is CodeBuddy's user-rejection-only fixed marker, verified by reverse-engineering the extension's tool state machine: it excludes system auto-cancels (UNFINISHED TOOL / MalformedToolArgs) and interrupt residue, which also land on cancelled but never carry this string.
  • toolError — a tool blob whose tool-result has isError === true. Executed and rejected tools both carry isError === false, so only genuine execution failures count — matching Claude's "is_error=true and not a reject" semantics.

Both counts are de-duplicated per callId. interrupt stays 0 — CodeBuddy has no on-disk marker for it (an interrupted turn's residual cancelled tools carry no user-rejection string, so they are correctly not miscounted as rejects).

Nothing downstream changes: computeSmartScore, the weights, the threshold, and the Claude/Cursor branches are untouched. The toolReject/toolError fields already existed on TranscriptScanResult — this only fills them in for CodeBuddy.

Safety

Parsing is fully defensive: readdir and every per-blob stat/readFile/JSON.parse/shape access are guarded, any single-blob failure is skipped, and the function never throws — the Stop hook is never destabilized. Blob count is capped (CODEBUDDY_BLOB_MAX_COUNT) to bound hook IO on pathological directories.

Test plan

  • npx tsc --noEmit — passes
  • npx vitest run src/__tests__/dashboard-collector.test.ts — 70 passed (7 new CodeBuddy cases: reject counting, error counting, isError=false not counted, per-callId dedup, mixed session, missing messages dir, malformed-blob resilience)
  • npm run build — success
  • Real-data end-to-end: against a real CodeBuddy session where a rm command was rejected → {interrupt:0, toolReject:1, toolError:2} (reject precisely captured, signals not cross-counted)
  • Negative control: 20 ordinary CodeBuddy sessions → all 0/0 (zero false positives)

🤖 Generated with Claude Code

m0Nst3r873 and others added 2 commits August 26, 2026 17:33
… transcripts

readCodebuddyIndexOnce hardcoded interrupt/toolReject/toolError to 0
because CodeBuddy's index.json skeleton carries no tool results, so
CodeBuddy sessions never scored friction and never triggered the
share-learnings hint (discussion Tencent#328).

Tool results live in the sibling messages/*.json blobs instead. Add
scanCodebuddyBlobs to read them once (before the token-flush retry loop,
so IO isn't amplified) and merge two signals into the scan result:

- toolReject: an assistant blob whose extra.toolStatus[callId] is
  cancelled with result.errorMessage containing "User rejected this
  command" — CodeBuddy's user-rejection-only marker, which excludes
  system auto-cancels and interrupt residue.
- toolError: a tool blob whose tool-result has isError === true.

Both are de-duplicated per callId. interrupt stays 0 (CodeBuddy has no
on-disk marker for it). Parsing is fully defensive: any single-blob
failure is skipped and the function never throws, so the Stop hook is
never destabilized. Blob count is capped to bound hook IO.

Verified end-to-end against a real rejection session (1 reject / 2
errors) and 20 ordinary sessions (0/0, no false positives).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Stop hook runs dashboardReportHandler (detached background process) and
contributeCheckHandler (foreground) concurrently. The background process
scans the transcript and writes the interventions snapshot into
events.jsonl; the foreground contribute-check reads that snapshot to score
friction. The foreground read routinely wins the race and sees an
un-flushed (empty) friction snapshot, so friction-driven sessions never
score above threshold and the share-learnings hint is silently dropped.
On CodeBuddy the token-flush retry delays the background write further,
making the miss near-deterministic.

Fix: contribute-check now scans the transcript itself (scanTranscriptStop)
for interrupt/toolReject/toolError instead of depending on the
possibly-unwritten Stop event. correction still derives from events
(a stop->prompt_submit pattern, unaffected by the race). Fully backward
compatible: without a transcript path the original events-snapshot path
runs unchanged.

Add a frictionOnly fast path so the foreground caller skips CodeBuddy's
~1.75s token-flush retry loop — friction comes from the one-shot blob scan
and needs no token data, saving foreground hook budget (4.5s cap).

Verified end-to-end against the real CodeBuddy session that first exposed
the race (3f1ac58e): friction now reads {toolReject:1, toolError:2},
enough to cross the hint threshold.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@m0Nst3r873
m0Nst3r873 force-pushed the feature/codebuddy-friction-signals branch from 9002f23 to ef4f91f Compare August 27, 2026 02:39
@m0Nst3r873

Copy link
Copy Markdown
Collaborator Author

Update: added a second commit for a race that blocked the hint end-to-end

Real-device testing on CodeBuddy surfaced a follow-on bug that the first commit alone didn't fully resolve, so this PR now carries two commits (rebased onto latest main):

  1. feat(dashboard): extract toolReject/toolError from CodeBuddy transcripts (original).
  2. fix(contribute): read friction from the transcript to avoid a Stop-event race.

The race

At Stop, dashboardReportHandler (a detached background process) scans the transcript and writes the interventions snapshot into events.jsonl, while contributeCheckHandler (foreground) reads that snapshot to score friction. The foreground read routinely wins the race and sees an un-flushed, all-zero friction snapshot — so a friction-heavy session scores below threshold and the share-learnings hint is silently dropped. On CodeBuddy the token-flush retry delays the background write further, making the miss near-deterministic.

Confirmed on a real CodeBuddy session: the Stop event recorded {toolReject:1, toolError:1} (proving commit 1's extraction works), yet contribute-check scored it with reject=0 and dropped the hint.

The fix

contribute-check now scans the transcript itself (scanTranscriptStop) for interrupt/toolReject/toolError instead of depending on the possibly-unwritten Stop event. correction still derives from events (a stop→prompt_submit pattern, unaffected). Fully backward compatible — without a transcript path the original events-snapshot path runs unchanged.

A frictionOnly fast path lets the foreground caller skip CodeBuddy's ~1.75s token-flush retry (friction comes from the one-shot blob scan and needs no token data), keeping the foreground Stop hook well inside its ~4.5s budget.

Verification

  • Full suite green (2238 tests), incl. a new race regression test (Stop event all-zero + transcript with a reject → toolReject === 1, hint fires) and a backward-compat test.
  • End-to-end against the real session that first exposed the race (3f1ac58e): friction now reads {toolReject:1, toolError:2} — enough to cross the hint threshold.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant