Run StaticIndex::compute over multiple threads#22530
Open
nicolas-guichard wants to merge 5 commits into
Open
Conversation
rustbot
added
the
S-waiting-on-review
nicolas-guichard
commented
Jun 5, 2026
| }; | ||
|
|
||
| for (def, token) in tokens { | ||
| lsif.add_token(def, token); |
Contributor
Author
There was a problem hiding this comment.
I'm not too happy about this: ideally lsif_contains_generated_constant would be order-independent but is it worth the time?
Veykril
reviewed
Jun 6, 2026
Veykril
reviewed
Jun 6, 2026
This comment has been minimized.
This comment has been minimized.
TokenIds are auto-increment identifiers for Definitions (which would be better named Symbol according to its docstring). This auto-increment property creates shared state that makes a TokenStore diffucult to share accross threads and merging multiple StaticIndex instances non- trivial. Instead we can just use Definition itself which is Hash and Copy. This changes the order tokens are iterated on in lsif.rs:334.
…ysis Each thread will have its own Analysis clone, and return a partial StaticIndex to the main thread. Thus StaticIndex must survive Analysis.
Thanks to the 4 previous commits, everything is in place to start multiple threads in StaticIndex::compute. Each thread takes care of the files whose index in the precomputed list is congruent to the thread index modulo the thread count and generates a partial StaticIndex. All those StaticIndex instances are then simply concatenated together. On my laptop with 16 threads this brings the time required to generate an SCIP index of Firefox from 80 seconds with 160% CPU usage down to 30 seconds, with 600% CPU usage.
When multi-threading StaticIndex::compute, we want the list of files to be computed only once and the files to be split between threads.
When multi-threading StaticIndex::compute, each thread will get its own Analysis clone.
nicolas-guichard
force-pushed
the
push-xqywvtlynoxs
branch
from
23b27c3 to
a7252fb
Compare
Collaborator
|
This PR was rebased onto a different master commit. Here's a range-diff highlighting what actually changed. Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Each thread takes care of the files whose index in the precomputed list is congruent to the thread index modulo the thread count and generates a partial StaticIndex. All those StaticIndex instances are then simply concatenated together.
On my laptop with 16 threads this brings the time required to generate an SCIP index of Firefox from 80 seconds with 160% CPU usage down to 30 seconds, with 600% CPU usage.
Fixes #18140