Skip to content

fix: stream JSONL reads so token widgets work on large transcripts - #551

Open
LZong-tw wants to merge 1 commit into
sirmalloc:mainfrom
LZong-tw:fix/stream-jsonl-token-metrics
Open

fix: stream JSONL reads so token widgets work on large transcripts#551
LZong-tw wants to merge 1 commit into
sirmalloc:mainfrom
LZong-tw:fix/stream-jsonl-token-metrics

Conversation

@LZong-tw

Copy link
Copy Markdown

Summary

Fixes #550.

Built-in tokens-input / tokens-output / tokens-total widgets showed In: 0 Out: 0 Total: 0 for long-lived Claude Code sessions once the session transcript JSONL exceeded Node's maximum string length (~512MB / 0x1fffffe8).

Root cause

readJsonlLines did:

const content = await readFile(filePath, 'utf-8');

On an ~844MB transcript this throws:

Error: Cannot create a string longer than 0x1fffffe8 characters

getTokenMetrics caught the error and returned all zeros. Token widgets prefer context.tokenMetrics even when zeroed, so they never fell back to context_window totals. Session cost still worked because it comes from statusline stdin, not the transcript scan.

Changes

  • Replace whole-file readFile / readFileSync with streaming line iterators:
    • async: fs.createReadStream + readline
    • sync: chunked readSync with Buffer leftover handling (UTF-8 safe across chunk boundaries)
  • readJsonlLines / readJsonlLinesSync now assemble lines from those iterators (fixes speed metrics, compaction, session clock, etc. for the same class of failure)
  • getTokenMetrics and getSessionDuration iterate lines without materializing the full file as one string
  • Tests for streaming/CRLF/empty lines and multi-line aggregation; existing token-metric tests still pass

Verification

npx vitest run src/utils/__tests__/jsonl-lines.test.ts \
  src/utils/__tests__/jsonl-metrics.test.ts \
  src/widgets/__tests__/TokensWidgets.test.ts \
  src/utils/__tests__/jsonl-blocks.test.ts
# 41 passed

Against a real ~844MB production transcript (the session that reproduced #550):

Before After (this branch)
getTokenMetrics all zeros (string-length throw) totalTokens: 17_794_272_870 in ~7s
In/Out/Total widgets 0 / 0 / 0 non-zero

Test plan

  • Unit tests for streaming JSONL reader
  • Existing token/speed metric tests
  • Real multi-hundred-MB transcript no longer returns zeros
  • Reviewer: smoke statusline against a large session if available

Claude Code session transcripts can exceed Node's max string length
(~512MB). readJsonlLines used fs.readFile(..., 'utf-8'), which throws
Cannot create a string longer than 0x1fffffe8 characters. getTokenMetrics
swallowed that error and returned zeros, so In/Out/Total stayed at 0
while session-cost (from statusline stdin) still looked correct.

Stream line-by-line via createReadStream/readline (async) and chunked
Buffer reads (sync), and keep getTokenMetrics on the streaming path.

Fixes sirmalloc#550
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Token widgets show In/Out/Total: 0 when session transcript exceeds ~512MB

1 participant