[WIP] Fix batched silhouette score reduction aliasing

[csadorf]

Stores the row-wise minimum for batched silhouette scores in a separate device vector, preventing the reduction output from overwriting matrix elements that may still be read.

Validation

Validates the fix with the temporary BATCHED_SILHOUETTE_CONCURRENCY_TEST added in ef922834, originally e20bfffc before the rebase. The test has since been reverted.

As a negative control, reverting the fix while retaining the test caused the A100/CUDA 12.2 CI job to fail: 12 of 512 worker seeds produced 22 batched/non-batched mismatches above the 1e-4 tolerance. This confirms that the stress test detects the unfixed reduction-aliasing behavior in the target environment.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[csadorf]

/ok to test f8c9939

[csadorf]

/ok to test 6ec7aeb

[csadorf]

/ok to test 6ec7aeb

[csadorf]

/ok to test d24aff2

[csadorf]

/ok to test 77aaa23

[csadorf]

The negative-control run behaved as expected. With the presumed fix reverted, BATCHED_SILHOUETTE_CONCURRENCY_TEST failed on A100/CUDA 12.2: 12 of 512 worker seeds produced 22 batched/non-batched mismatches above the 1e-4 tolerance. The same test passed on L4/CUDA 13.

This confirms that the stress test detects the unfixed behavior in the target environment. The next step is to restore the fix and rerun the same matrix; the A100/CUDA 12.2 test should then pass.

The separate NN-descent recall failure on L4 appears unrelated.