Skip to content

NanoVDB CI on AWS CodeBuild GPUs - #2271

Open
swahtz wants to merge 27 commits into
masterfrom
ci/aws_codebuild
Open

NanoVDB CI on AWS CodeBuild GPUs#2271
swahtz wants to merge 27 commits into
masterfrom
ci/aws_codebuild

Conversation

@swahtz

@swahtz swahtz commented Aug 5, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

  • Build NanoVDB’s GCC/Clang Debug/Release matrix on GitHub CPU runners, then transfer the build artifacts to AWS CodeBuild for GPU execution.
  • Run CUDA, multi-GPU, and NanoVDB Python tests on 4× A10G GPUs while retaining CPU and OpenVDB test coverage.
  • Target A10G compute capability 8.6 and report GPU topology in CodeBuild logs.
  • Fix distributed-grid stream synchronization and reduce excessive memory use in the segmented-sort unit test.
  • Update GitHub Actions, remove unnecessary sudo usage.

Notable Changes to CI Setup

  • For nanovdb.yml Actions, this changes the trigger from pull_request to pull_request_target. This means that the job runs from the PR's target branch (instead of the PR's branch). The reason is that for security, Github Actions does not allow access to repository secrets from non-local/fork branches and we need these to authenticate with AWS. The code that runs in the PR actions will be from the PR's branch merged with the target but the big effect is that the job definition will have come from the target branch. This means that any PR that changes this job's definition will not be reflected in the CI run for that PR. This is an inherent characteristic of this trigger mode. If a Maintainer wants to test the effects of the candidate PR, the PR branch has to be local to the OpenVDB repo and can be triggered manually in the 'Actions' tab to test run from the PR branch.
  • The job/task description that runs on the AWS GPU machines is now written in AWS CodeBuild's Buildspec format. The CodeBuild job is started from the .github/workflows/nanovdb.yaml Action and the CodeBuild tasks that run the tests are defined in the ci/buildspec_nanovdb.yml.

Test plan

  • CPU matrix and nanovdb-lite pass.
  • Build artifacts download and restore successfully in CodeBuild.
  • CUDA and Python GPU tests pass.
  • GCC Debug/Release and Clang Release GPU jobs pass.

swahtz added 16 commits August 5, 2026 13:14
@swahtz
Add GPU workflow
612afc8 
Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
@swahtz
Add push trigger
30a86c4 
Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
@swahtz
Upgrade credentials version
dbbc3cf 
Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
@swahtz
Add a buildspec with trivial work
dbb6e04 
Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
@swahtz
Fix project name
20ce219 
Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
@swahtz
Fix buildspec
026b9aa 
Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
@swahtz
Fix buildspec
9ae31a5 
Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
@swahtz
Switch to 2026 image; add tree install
fd2a48e 
Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
@swahtz
nvidia-smi topo
52adef1 
Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
@swahtz
Full NanoVDB Tests
8a527f2 
Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
@swahtz
NanoVDB CI fixes
5b56774 
Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
@swahtz
Upgrade GH actions versions
61b0fae 
Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
@swahtz
Changed ManyTiles_CudaPointsToGrid to:
76fd981 
Generate 64 deterministic upper-node tiles.
Put 64 voxels in each tile.
Still trigger the >= 32 segmented-sort path.
Assert exactly 64 upper nodes were created.
Reduce estimated node memory to about 18.6 MiB.

Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
@swahtz
fix(nanovdb): synchronize distributed grid initialization
46b9b82 
Record the root initialization event on its CUDA stream to prevent dependent GPU work from racing, and improve active voxel count diagnostics.

Moved buildspec inside ci/

Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
@swahtz
test(nanovdb): expand CodeBuild GPU coverage
72c0dd0 
Run the NanoVDB Python suite on GPU workers and expose the selected CodeBuild project and fleet configuration for easier CI diagnosis.

Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
@swahtz
job cleanup
6d7ff5f 
Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
@swahtz swahtz added the nanovdb label Aug 5, 2026
swahtz and others added 7 commits August 6, 2026 08:49
@swahtz
Fix checkout and statements to refelct pull_request_target
27f141e 
Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
@swahtz
mGPU Diagnostics
4921d36 
Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
@swahtz
Upload artifacts directly to Github
196dbf3 
Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
@swahtz
Pin fix
dd2526e 
Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
@swahtz
IOMMU checks
d614dce 
Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
@harrism @claude @swahtz
NanoVDB CI: repeat the mgpu suite until failure
5ac5538 
The remaining DenseLeaf undercount is an intermittent race -- the same
code failed and passed clang-Debug on consecutive days -- so a single
green run says nothing. Run the mgpu tests up to twenty times after the
main pass, stopping at the first failure; at seventeen seconds per
iteration the worst case adds about six minutes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Mark Harris <mharris@nvidia.com>
@harrism @claude @swahtz
NanoVDB: report per-device counts after the distributed count phase
b3d4644 
Diagnostic for the intermittent DenseLeaf undercount, whose failing
value is exactly one device's quarter of the voxels. The counts are
printed once they are final -- after the full per-device stream
synchronize -- so a failing run shows directly whether a stripe was
already lost upstream of aggregation or survived counting and was lost
downstream.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Mark Harris <mharris@nvidia.com>
swahtz added 4 commits August 7, 2026 14:20
@swahtz
NanoVDB: assign each upper-node tile to a single GPU in DistributedPo…
67ff8ca 
…intsToGrid

The distributed builder distributes work by snapping device boundaries to
upper-node (tile) boundaries, since each tile subtree is counted and built
independently on one device. The previous pairwise rebalance could not
consolidate a tile whose points span three or more devices (a fully-interior
device lies entirely within the tile), so such a tile stayed split. Multiple
devices then concurrently registered and cleared/set the same leaf's value
mask with no cross-device ordering, intermittently losing updates and
undercounting active voxels (e.g. the single dense-leaf case collapsing to
one device's stripe).

Replace the pairwise TransformReduce + rebalance kernels with a global,
monotonic host-side boundary snap over the globally-sorted keys. Every tile
is now owned by exactly one device (interior devices may be left empty),
eliminating cross-device writes to shared nodes. Remove the now-unused
EqualityIndicator, left/rightRebalanceKernel, and the transformReduce/rebalance
event arrays.

Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
@swahtz
NanoVDB: add multi-GPU regression tests for shared-tile ownership
de2ab11 
Add deterministic tests covering the DistributedPointsToGrid path where a
single upper-node tile is split across all devices:

- SingleUpperNode: many voxels within one 4096^3 tile spanning many
  lower/leaf nodes; asserts the exact active-voxel count so a reintroduced
  cross-device race is caught deterministically rather than probabilistically.
- MatchesSingleGpu: cross-checks topology and voxel occupancy against the
  trusted single-GPU voxelsToGrid builder on a tile-splitting input.
- FewerVoxelsThanDevices: guards the empty-stripe edge case.

Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
@swahtz
NanoVDB: drop FewerVoxelsThanDevices multi-GPU test
76c1643 
The test exercised inputCount < deviceCount, which triggers a pre-existing
limitation in radixSortAsync (the initial per-device sort runs on a zero-count
segment and CUB returns "invalid device ordinal"). That path is unrelated to
the shared-tile ownership fix, and the empty interior-stripe behavior the fix
introduces is already covered by SingleUpperNode and MatchesSingleGpu.

Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
@swahtz
NanoVDB: remove multi-GPU diagnostics now that the race is fixed
fb5a6cc 
Drop the per-device stripe/voxel-count fprintf in DistributedPointsToGrid and
the "--repeat until-fail:20" mgpu loop in the CodeBuild buildspec. Both were
temporary scaffolding for diagnosing the shared-tile race, which is now fixed
and covered by dedicated regression tests.

Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kmuseth kmuseth Awaiting requested review from kmuseth kmuseth is a code owner
@apradhana apradhana Awaiting requested review from apradhana apradhana is a code owner
@jmlait jmlait Awaiting requested review from jmlait jmlait is a code owner
@Idclip Idclip Awaiting requested review from Idclip Idclip is a code owner
@danrbailey danrbailey Awaiting requested review from danrbailey danrbailey is a code owner
@richhones richhones Awaiting requested review from richhones richhones is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

nanovdb

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@swahtz @harrism