-
Notifications
You must be signed in to change notification settings - Fork 195
[FEA] Multi-node Out of Core Streaming KMeans API #2066
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
tarang-jain
wants to merge
198
commits into
NVIDIA:main
Choose a base branch
from
tarang-jain:mnmg-streaming
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
198 commits
Select commit
Hold shift + click to select a range
66d7fd3
combine impls
tarang-jain 07707af
Multi-GPU Batched KMeans
viclafargue efc270f
Merge branch 'main' into mg-batched-kmeans
viclafargue 0a09e6f
rm inertia_check
tarang-jain 99a5730
change to warning
tarang-jain a077406
style
tarang-jain d659875
add init_size param
tarang-jain ec2e8b7
Merge branch 'main' into combine-batch
tarang-jain 03a6473
docs
tarang-jain 42a8d9d
Merge branch 'combine-batch' of https://github.com/tarang-jain/cuvs i…
tarang-jain 86af2fa
rm direct cuda api calls
tarang-jain d4e4e2c
std::swap instead of raft::copy
tarang-jain 0819af5
cache batch norms
tarang-jain e0f079c
centroid norms can also be cached per iteration
tarang-jain c2f7390
mg n_iter
tarang-jain b9c3102
pre-commit
tarang-jain e3956c1
do not break c abi
tarang-jain 986d78a
Merge branch 'main' into combine-batch
tarang-jain 7197b71
cluster_cost on device
viclafargue 84ab315
Updated testing
viclafargue 47d4b94
templating
viclafargue a8e1d26
Merge branch 'main' into combine-batch
tarang-jain 384d054
fix checkWeight
tarang-jain 455b286
merge upstream:
tarang-jain 5462809
Merge branch 'combine-batch' of https://github.com/tarang-jain/cuvs i…
tarang-jain 6ba759c
fix compilation
tarang-jain e76eaac
rel_tol
tarang-jain afbefdf
pass workspace
tarang-jain e62a63c
Merge branch 'combine-batch' of https://github.com/tarang-jain/cuvs i…
tarang-jain e4f08bf
style
tarang-jain 6e4a8f0
Merge branch 'main' of https://github.com/rapidsai/cuvs into combine-…
tarang-jain 4a8a85c
do not use batch scratch space; rm update_centroids
tarang-jain bbf2a9f
move the debug log
tarang-jain 410092c
add new suffixed param struct
tarang-jain c515c1e
address pr reviews
tarang-jain e8e63ab
fix docstring
tarang-jain 30c457c
fix wt_sum warning
tarang-jain ab96623
rm deprecationwarning and instead add FutureWarning:=
tarang-jain 269f23c
unweighted to never materialize batch weights
tarang-jain 80a22ca
add cpp tests
tarang-jain ac06b05
update cpp tests
tarang-jain 855624a
Merge branch 'main' into mg-batched-kmeans
viclafargue 0a6748d
refactor
viclafargue 7055272
rename to mnmg_fit
viclafargue 0569340
revert batch norms cache
tarang-jain 8cac63a
increase zero cost threshold
tarang-jain f6df4ae
apply cuda event plus re-add h_norm_cache
tarang-jain 9fc74b1
rm cosine expanded stuff
tarang-jain dec3dc4
resolve merge conflicts
tarang-jain 0d030a2
change suffix of the params struct
tarang-jain b1c034e
replace 06 by 08, add todo and note
tarang-jain a482495
update to v2
tarang-jain 8ecfdc1
avoid stream sync inside weight sum
tarang-jain 1e1525e
Merge branch 'combine-batch' of https://github.com/tarang-jain/cuvs i…
tarang-jain ec22e07
empty
tarang-jain d2e410d
empty
tarang-jain b791c38
Merge branch 'main' into combine-batch
tarang-jain a05a006
new signatures with new struct
tarang-jain 73293cf
Merge branch 'combine-batch' of https://github.com/tarang-jain/cuvs i…
tarang-jain 880c7b9
Merge branch 'main' of https://github.com/rapidsai/cuvs into combine-…
tarang-jain e2035ec
revert change to calls in py and rust; add c tests
tarang-jain e28c200
Merge branch 'main' into combine-batch
tarang-jain 55bbdad
use to_dlpack
tarang-jain 9a9b8ee
cache device weights
tarang-jain a800b27
rm event
tarang-jain 3db8582
update names
tarang-jain c048352
rename
tarang-jain 2f968f8
rm docs
tarang-jain affe85a
empty
tarang-jain c6dea64
fix norm cache
tarang-jain 7dfab3e
revert changes to minClusterDistanceCompute
tarang-jain 7a383da
update tests to use mdspan instead of rmm
tarang-jain ce6c4b5
Merge branch 'main' into combine-batch
tarang-jain 5a06a44
Merge branch 'main' into combine-batch
tarang-jain 419619a
consolidate all unsigned commits
tarang-jain 2d716ae
rm diff
tarang-jain 066092b
allow batch sample weights
tarang-jain bbdd66d
Merge branch 'main' into mnmg-streaming
tarang-jain 12d682c
single partition becomes special case
tarang-jain 9e5e55c
Merge branch 'mnmg-streaming' of github.com:tarang-jain/cuvs into mnm…
tarang-jain 28cda6a
Merge branch 'combine-batch' into mg-batched-kmeans
viclafargue bfb5290
Addressing review
viclafargue add9db1
optimize convergence check
viclafargue 6c08a7b
Merge branch 'main' into mnmg-streaming
tarang-jain acbcd5a
Merge branch 'main' into mnmg-streaming
tarang-jain af606bc
Adressing review
viclafargue 41c66b8
Merge branch 'main' into mg-batched-kmeans
viclafargue f664c2c
results on all ranks for RAFT + small optimization
viclafargue 5430f42
Merge branch 'main' into mnmg-streaming
tarang-jain b2ab5bd
merge origin
tarang-jain bbdf521
Merge branch 'mnmg-streaming' of github.com:tarang-jain/cuvs into mnm…
tarang-jain 10e6def
Merge branch 'main' into mnmg-streaming
tarang-jain 2040145
reduce diff
tarang-jain 1828462
Merge branch 'mnmg-streaming' of github.com:tarang-jain/cuvs into mnm…
tarang-jain 5c5b8c8
Merge branch 'main' into mnmg-streaming
tarang-jain 05da5f3
rm prefetch
tarang-jain 90435c1
Merge branch 'mnmg-streaming' of github.com:tarang-jain/cuvs into mnm…
tarang-jain db41338
Merge branch 'main' into mnmg-streaming
tarang-jain 6c2c03d
reviews
viclafargue 7f6d664
Global sampling for init
viclafargue f8270e2
SNMG -> MNMG
viclafargue bbf0302
Merge branch 'main' into mg-batched-kmeans
viclafargue a14a6bc
adding asserts
viclafargue 7b54a42
consume new init
tarang-jain d86b8b4
Merge branch 'mnmg-streaming' of github.com:tarang-jain/cuvs into mnm…
tarang-jain 6e11f67
reduce diff
tarang-jain 9f5b6e5
Merge branch 'main' into mnmg-streaming
tarang-jain aaef638
rm unnecessary functions
tarang-jain 920a460
Merge branch 'main' into mnmg-streaming
tarang-jain 548d7db
rm accessor templates for now
tarang-jain 9f3a486
Merge branch 'mnmg-streaming' of github.com:tarang-jain/cuvs into mnm…
tarang-jain c93f248
Merge branch 'main' of https://github.com/rapidsai/cuvs into mnmg-str…
tarang-jain 51fbf6c
merge upstream
tarang-jain d327569
cleanup; re-add device side overload
tarang-jain b5e66a3
re-instate removed docs
tarang-jain a636188
rm extra fit funcs
tarang-jain d3cafed
Merge branch 'release/26.06' into mnmg-streaming
tarang-jain 1b547f4
Merge branch 'mnmg-streaming' of github.com:tarang-jain/cuvs into mnm…
tarang-jain 72cfd43
cleanup
tarang-jain 4d25e95
rm scaled_weights_cache
tarang-jain 81155e6
rm unnecessary new types
tarang-jain 85522aa
rm unused helper
tarang-jain 00336b5
rm unnecessary stream sync
tarang-jain 6585866
rm unnecessary lambda
tarang-jain aa6f28e
cleanup impl
tarang-jain 178a7e7
rm unnecessary has_data guards
tarang-jain 713bc7c
rm global_n host scalar
tarang-jain c576d8f
fixes
tarang-jain a401a0e
Merge branch 'release/26.06' into mnmg-streaming
tarang-jain 8102596
fuse with in-memory impl
tarang-jain 00d0adb
Merge branch 'mnmg-streaming' of github.com:tarang-jain/cuvs into mnm…
tarang-jain caefd53
style
tarang-jain 8f6f83d
fix compilation
tarang-jain d88a991
Merge branch 'release/26.06' into mnmg-streaming
tarang-jain 1b57b74
mg tests first commit
tarang-jain 7bac418
Merge branch 'mnmg-streaming' of github.com:tarang-jain/cuvs into mnm…
tarang-jain 9851017
merge upstream
tarang-jain f572877
update cmakelists
tarang-jain edaa7e7
merge upstream
tarang-jain 588bb6a
rm batched tests
tarang-jain ad180ed
Merge branch 'main' into mnmg-streaming
tarang-jain 72cc34b
rm unnecessary test stream sycns
tarang-jain ed50703
reset bs; assertion
tarang-jain 28f6036
Merge branch 'mnmg-streaming' of github.com:tarang-jain/cuvs into mnm…
tarang-jain a811c56
rm has_data flag
tarang-jain 95f334c
Merge branch 'main' into mnmg-streaming
tarang-jain d176314
fix export
tarang-jain 1db9e02
Merge branch 'mnmg-streaming' of github.com:tarang-jain/cuvs into mnm…
tarang-jain 089e970
avoid pinned scalar;get_nccl_comms before omp
tarang-jain 6cc895c
use root from macro
tarang-jain 4abe6f2
avoid copy and rank alloc with initarray
tarang-jain ebf188a
Merge branch 'main' of https://github.com/rapidsai/cuvs into mnmg-str…
tarang-jain 785e4a3
fix compilation; guardrail MG CMake flag
tarang-jain 9a526c8
get n_features from centroids
tarang-jain f08e581
add sigs to header
tarang-jain 51efb42
Merge branch 'main' into mnmg-streaming
tarang-jain 3e3cac7
Merge branch 'main' into mnmg-streaming
tarang-jain 7ffae6d
Merge branch 'main' into mnmg-streaming
tarang-jain 447e136
put single mdspan in mg namespace
tarang-jain 50d0359
Merge branch 'mnmg-streaming' of https://github.com/tarang-jain/cuvs …
tarang-jain b1e7521
single reduce
tarang-jain f8a3503
re-add distributed kmeanspp
tarang-jain 272a9d5
revert differences from distributed init
tarang-jain 1f9fd9e
Merge branch 'main' into mnmg-streaming
tarang-jain ed705c6
optional allredice
tarang-jain c8135e5
Merge branch 'mnmg-streaming' of https://github.com/tarang-jain/cuvs …
tarang-jain cf5e831
Merge branch 'main' into mnmg-streaming
tarang-jain 8b6eab2
apply distributed init fixes
tarang-jain 100f7d6
Merge branch 'mnmg-streaming' of https://github.com/tarang-jain/cuvs …
tarang-jain 46d18d1
re-add host assertion for sum of weights
tarang-jain db23a72
remaining samples should be global
tarang-jain 1e7c119
doc updates
tarang-jain 116a6cf
add dask-like cpp test
tarang-jain ae284be
add warning to docs
tarang-jain d15cb28
Merge branch 'main' into mnmg-streaming
tarang-jain 653aac1
address pr reviews
tarang-jain 92588ba
Merge branch 'mnmg-streaming' of https://github.com/tarang-jain/cuvs …
tarang-jain ef60e3c
add weight check for sg device weights
tarang-jain 8831789
re-update the docs
tarang-jain 3f800a1
update docs
tarang-jain 32be863
undo fern changes
tarang-jain d0fa3a9
Merge branch 'main' into mnmg-streaming
tarang-jain 0b969c4
address reviews
tarang-jain 040e82b
Merge branch 'mnmg-streaming' of https://github.com/tarang-jain/cuvs …
tarang-jain d65287d
fix bcast
tarang-jain 3321926
style
tarang-jain de57f46
update docstring
tarang-jain 1a6bbb5
add oversampling test
tarang-jain e627af4
address reviews
tarang-jain fd19b43
Merge branch 'main' into mnmg-streaming
tarang-jain 993b813
Merge branch 'main' into mnmg-streaming
tarang-jain bd3e1de
Merge branch 'main' into mnmg-streaming
tarang-jain 40c77ca
Merge branch 'main' into mnmg-streaming
tarang-jain 079c216
Merge branch 'main' into mnmg-streaming
tarang-jain 7a38f62
Merge branch 'main' into mnmg-streaming
tarang-jain 6ada5f6
do not use nullptr for bcast
tarang-jain 8ce1ade
fix mnmg tests
tarang-jain 53ca72b
dependencies for devc
tarang-jain File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.