Skip to content

perf: reuse parsers across parallel graph builds#670

Open
Lum1104 wants to merge 1 commit into
tirth8205:mainfrom
Lum1104:feat/accelerate-multilang-builds
Open

perf: reuse parsers across parallel graph builds#670
Lum1104 wants to merge 1 commit into
tirth8205:mainfrom
Lum1104:feat/accelerate-multilang-builds

Conversation

@Lum1104

@Lum1104 Lum1104 commented Jul 19, 2026

Copy link
Copy Markdown

Linked issue

Related to #189 and the parser-load hardening merged in #459.

What & why

Parallel full and incremental builds called CodeParser(...) once per file. That discarded the parser instance cache on every task and repeatedly paid language-pack/parser initialization inside each worker. On a mixed-language checkout of this repository, the default 8-process raw build took 36.23 seconds even though serial parsing took 7.71 seconds.

This change keeps one parser per worker and repository in threading.local() storage:

  • process workers reuse parser state across their assigned files;
  • thread workers retain isolated mutable parser state;
  • changing repository roots invalidates the worker-local parser;
  • executor selection, worker-count policy, error handling, and serial SQLite writes remain unchanged.

The worker-local design also applies to incremental builds because both build paths share _parse_single_file.

Impact and benchmark

Same 215-file mixed-language repository, fresh project-local data directory, --skip-postprocess, Python 3.14.2 on a 10-core Apple Silicon host:

Configuration Before After
8 process workers (default) 36.23 s 7.11 s
4 process workers 5.59 s 6.32 s

The default path improves by about 5.1x. The two post-change worker counts produced identical graphs: 4,420 nodes and 34,162 edges. I intentionally left worker-count heuristics unchanged; the measured regression is parser lifecycle, while a global worker cap needs broader repository-size and platform data.

How it was tested

uv run pytest tests/test_incremental.py::TestParseExecutorSelection tests/test_incremental.py::TestParallelParsing tests/test_mcp_stdio_shutdown.py -q
# 10 passed

CRG_PARSE_EXECUTOR=thread uv run pytest tests/test_incremental.py::TestParallelParsing::test_parallel_build_produces_same_results -q
CRG_PARSE_EXECUTOR=process uv run pytest tests/test_incremental.py::TestParallelParsing::test_parallel_build_produces_same_results -q
# 1 passed for each executor

uv tool run ruff check code_review_graph/incremental.py tests/test_incremental.py
# All checks passed

uv tool run mypy code_review_graph/incremental.py --ignore-missing-imports --no-strict-optional
# Success: no issues found

uv run pytest -q
# 1963 passed, 5 skipped, 2 xpassed, 1 unrelated failure

The full-suite failure is pre-existing on Python 3.14/macOS: test_windows_server_still_prewarms_before_mcp_run monkeypatches asyncio.WindowsSelectorEventLoopPolicy after setting sys.platform to win32, which raises inside Python 3.14 asyncio before this code path is reached.

Checklist

  • Tests added for new functionality
  • All tests pass: one unrelated Python 3.14 platform-emulation failure documented above
  • Linting passes
  • Type checking passes for the changed production module
  • Lines are at most 100 characters
  • Docstring and implementation comments updated for the worker lifecycle behavior

@Lum1104
Lum1104 marked this pull request as ready for review July 19, 2026 15:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant