perf: reuse parsers across parallel graph builds#670
Open
Lum1104 wants to merge 1 commit into
Open
Conversation
6 tasks
Lum1104
marked this pull request as ready for review
July 19, 2026 15:48
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Linked issue
Related to #189 and the parser-load hardening merged in #459.
What & why
Parallel full and incremental builds called
CodeParser(...)once per file. That discarded the parser instance cache on every task and repeatedly paid language-pack/parser initialization inside each worker. On a mixed-language checkout of this repository, the default 8-process raw build took 36.23 seconds even though serial parsing took 7.71 seconds.This change keeps one parser per worker and repository in
threading.local()storage:The worker-local design also applies to incremental builds because both build paths share
_parse_single_file.Impact and benchmark
Same 215-file mixed-language repository, fresh project-local data directory,
--skip-postprocess, Python 3.14.2 on a 10-core Apple Silicon host:The default path improves by about 5.1x. The two post-change worker counts produced identical graphs: 4,420 nodes and 34,162 edges. I intentionally left worker-count heuristics unchanged; the measured regression is parser lifecycle, while a global worker cap needs broader repository-size and platform data.
How it was tested
The full-suite failure is pre-existing on Python 3.14/macOS:
test_windows_server_still_prewarms_before_mcp_runmonkeypatchesasyncio.WindowsSelectorEventLoopPolicyafter settingsys.platformtowin32, which raises inside Python 3.14 asyncio before this code path is reached.Checklist