perf(cluster): call graspologic_native.leiden() directly, skip graspologic's import chain - #3104
Open
Mohammad-Palla wants to merge 1 commit into
Open
perf(cluster): call graspologic_native.leiden() directly, skip graspologic's import chain#3104Mohammad-Palla wants to merge 1 commit into
Mohammad-Palla wants to merge 1 commit into
Conversation
…logic's import chain graspologic's own package import (graspologic.layouts -> umap -> pynndescent -> numba JIT) costs 7-19s per process and has nothing to do with the leiden() call _partition() actually uses; graspologic_native (the PyO3 binding it wraps) imports in ~0.5ms. Measured on the Bun corpus (161k nodes / 336k edges), matched load: cluster stage 27.3-28.6s -> 17.9s (~36-37% faster), total build 155.4-159.8s -> 146.7s (~6-8% faster). Verified correctness by feeding the identical graph to both code paths in-process (exact partition match, Adjusted Rand Index 1.0) and by confirming a full test-suite pass with no regressions.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
_partition()currently reachesgraspologic.partition.leiden()through the fullgraspologicpackage import, which eagerly pulls ingraspologic.layouts→umap→pynndescent→ numba JIT compilation — 7-19s per process, for a layout module the clustering path never calls.graspologic_native(the PyO3/Rust bindinggraspologic.partition.leiden()itself wraps) imports in ~0.5ms. This adds a_native_leiden()adapter that calls it directly, falling back to the existinggraspologic.partition.leidenpath (then NetworkX Louvain) ifgraspologic_nativeisn't installed or the graph shape isn't supported.extractors/resolution.py's_disambiguate_colliding_node_idsruns before the graph is built), so the string-collision casegraspologic's_IdentityMapperguards against can't occur here; the adapter keeps a defensive check anyway.Measured (Bun corpus, 161k nodes / 336k edges, golden-cache-restored before each run)
clusterstageCorrectness verified, not assumed
tests/test_cluster.py: 11/11.graspologic_nativeitself being called identically by both, not something this change introduces. Confirmed separately that Graphify's own extraction pipeline never produces a negative/zero/NaN edge weight (every extractor hardcodesweight: 1.0), so that pre-existing panic risk isn't reachable through normal use.Note on run-to-run variance
Separate full-pipeline invocations show community-label drift between runs (Adjusted Rand Index ≈ 0.73-0.75), but this is pre-existing and identical on both code paths — traced to the already-filed #2817 / #1667 (hash-seed-sensitive iteration in
cluster()'s post-processing splitting pass, downstream of_partition()), not introduced or worsened by this change.Test plan
pytest tests/test_cluster.py— 11/11 passed