fix: stream JSONL reads so token widgets work on large transcripts - #551
Open
LZong-tw wants to merge 1 commit into
Open
fix: stream JSONL reads so token widgets work on large transcripts#551LZong-tw wants to merge 1 commit into
LZong-tw wants to merge 1 commit into
Conversation
Claude Code session transcripts can exceed Node's max string length (~512MB). readJsonlLines used fs.readFile(..., 'utf-8'), which throws Cannot create a string longer than 0x1fffffe8 characters. getTokenMetrics swallowed that error and returned zeros, so In/Out/Total stayed at 0 while session-cost (from statusline stdin) still looked correct. Stream line-by-line via createReadStream/readline (async) and chunked Buffer reads (sync), and keep getTokenMetrics on the streaming path. Fixes sirmalloc#550
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #550.
Built-in
tokens-input/tokens-output/tokens-totalwidgets showed In: 0 Out: 0 Total: 0 for long-lived Claude Code sessions once the session transcript JSONL exceeded Node's maximum string length (~512MB /0x1fffffe8).Root cause
readJsonlLinesdid:On an ~844MB transcript this throws:
getTokenMetricscaught the error and returned all zeros. Token widgets prefercontext.tokenMetricseven when zeroed, so they never fell back tocontext_windowtotals. Session cost still worked because it comes from statusline stdin, not the transcript scan.Changes
readFile/readFileSyncwith streaming line iterators:fs.createReadStream+readlinereadSyncwith Buffer leftover handling (UTF-8 safe across chunk boundaries)readJsonlLines/readJsonlLinesSyncnow assemble lines from those iterators (fixes speed metrics, compaction, session clock, etc. for the same class of failure)getTokenMetricsandgetSessionDurationiterate lines without materializing the full file as one stringVerification
Against a real ~844MB production transcript (the session that reproduced #550):
getTokenMetricstotalTokens: 17_794_272_870in ~7sTest plan