Skip to content

Continue merging main into 13.4.x - #2641

Draft
rwgk wants to merge 32 commits into
NVIDIA:13.4.x-merge-main-2026-08-10+0131from
rwgk:13.4.x-merge-main-cont
Draft

Continue merging main into 13.4.x#2641
rwgk wants to merge 32 commits into
NVIDIA:13.4.x-merge-main-2026-08-10+0131from
rwgk:13.4.x-merge-main-cont

Conversation

@rwgk

@rwgk rwgk commented Aug 15, 2026

Copy link
Copy Markdown
Contributor

Description

Follow-up to #2588.

Merge the newer public main history into the 13.4.x release line on top of the baseline established by #2588.

This continuation also repairs merge accidents exposed by the initial CI run:

  • temporarily skips the pixi CUDA-version check because CUDA 13.4 Conda packages are not yet available;
  • regenerates the CUDA 13.4.0 bindings with the current cybind generator fixes while preserving the 13.4 API surface;
  • avoids aggregate CUmemLocation initialization now that CUDA 13.4 adds the localized member; and
  • regenerates the merged cuda.core stubs.

The 13.4.x-merge-main-cont branch is intentionally also pushed to the upstream repository so it can serve as the base for another main-merge continuation stacked on top of this PR.

Testing

  • CUDA 13.4 bindings/core build runbook
  • CUDA 13.4 pathfinder, bindings default-stream and PTDS, bindings Cython, cuda.core, and cuda.core Cython test runbooks
  • generated-file seals and cuda.core stub generation
  • pre-commit suite, including Ruff, mypy, actionlint, Cython lint, YAML, and secret scanning

Checklist

  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

gmanal and others added 30 commits August 11, 2026 14:11
…VIDIA#2589)

Call the centrally maintained NVIDIA/security-workflows security suite rather
than wiring each scan separately: one pinned reference runs the Pulse secret
scan and CodeQL SAST, both explicitly enabled.

Replace .github/workflows/codeql.yml with the suite's SAST scan. Both publish
code scanning results under the category /language:python, so keeping the local
workflow would put two analyses on every commit that overwrite each other's
alerts. The suite performs the same analysis: python, build-mode none,
security-extended queries, on ubuntu-latest.
* fix(cuda.core): avoid truncating graph queries

* perf(cuda.core): retain adjacency stack buffer

* test(cuda.core): cover large predecessor graph queries

Verify exact edge identities so graph query regressions cannot pass through count-only checks.

---------

Co-authored-by: Andy Jost <ajost@nvidia.com>
* Fix Windows binary utility discovery on Arm64

* Clarify binary utility search order

* Expand standalone installation documentation

* Align standalone search step comments

* Preserve literal Nsight launcher lookup

* Cover Windows binary discovery fallbacks

* Document Windows architecture selection

* Harden Windows Arm64 utility discovery

* Fix Windows pre-commit checks

* Fix CUDA path precedence documentation

* Document Windows binary utility discovery

---------

Co-authored-by: Michael Wang <isVoid@users.noreply.github.com>
Co-authored-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com>
…VIDIA#2493)

* Migrate _static_libs finders from os.path to pathlib

Part 2 of the series proposed in NVIDIA#2410, following the same conversion
style as part 1 (NVIDIA#2489).

Path construction, joining, and filesystem predicates in
find_static_lib.py and find_bitcode_lib.py now go through pathlib.Path
instead of os.path string manipulation. Both modules keep importing os
solely for os.environ.get("CONDA_PREFIX").

Compatibility is preserved: every entry point still accepts str, and
every function that documents or returns str still returns str. Path is
used strictly as the internal representation and converted back with
str() at each return, so LocatedStaticLib.abs_path, LocatedBitcodeLib
.abs_path, find_static_lib() and find_bitcode_lib() are unchanged in
both type and value. No signature changes.

Signed-off-by: LeSingh1 <sshaurya914@gmail.com>

* Return Path from the _static_libs internals

Follow-up to the review feedback on NVIDIA#2489: the str-compatibility constraint
applies only to the public API.

The try_* methods and _no_such_file_in_dir now work in Path throughout. str()
is applied once, where abs_path is stored on the public LocatedStaticLib and
LocatedBitcodeLib. The relative-path constants go from os.path.join(...) to
forward-slash literals, matching how site_packages_dirs is already written in
the same dicts; Path normalizes the separator on Windows.

One behavior change: a CUDA_PATH or CONDA_PREFIX containing redundant
separators ("//", "/.") now produces a normalized abs_path, because Path
collapses them. Differential fuzzing against the pre-revision code (16k lookups
over randomized trees, comparing located paths and full error text) shows no
other difference, and none at all when those variables are free of redundant
separators.

Signed-off-by: LeSingh1 <sshaurya914@gmail.com>

---------

Signed-off-by: LeSingh1 <sshaurya914@gmail.com>
Co-authored-by: Michael Droettboom <mdboom@gmail.com>
* chore: fix Apache-2.0 license notice and attribution gaps

An open-source license review flagged several Apache-2.0 compliance gaps.
This addresses three of them, plus the guard that let one class of them
through. Licensing metadata only; no logic changes.

Copyright notices (15 files)
  Two different defects that happened to share a symptom:
  - 14 files under cuda_bindings/examples/ carried a non-standard notice
    ("Copyright 2021-2026 NVIDIA Corporation.  All rights reserved.") with
    no (c), no SPDX-FileCopyrightText prefix, and the wrong entity casing.
  - toolshed/conda_create_for_pathfinder_testing.ps1 had the correct prefix
    and casing but was truncated before "& AFFILIATES. All rights reserved.".
  All now carry the canonical string. Years are preserved as found.

Header guard (toolshed/check_spdx.py)
  COPYRIGHT_REGEX made "& AFFILIATES. All rights reserved." optional, so a
  bare "NVIDIA CORPORATION" satisfied pre-commit. The suffix is now
  required. (The 14 example files were passing for a different reason:
  .spdx-ignore excludes cuda_bindings/examples/ entirely. That exclusion is
  left alone here, but the files now conform, so it can be dropped in a
  follow-up if desired.)

  Tightening the regex surfaced two pre-existing files whose notice was
  split or truncated -- cuda_core/cuda/core/_include/layout.hpp and
  toolshed/build_static_bitcode_input.py. Both are corrected so the
  mandated sentence appears verbatim on one line.

Third-party attribution (cuda_core/NOTICE)
  cuda/core/_include/aoti_shim.h is a vendored subset of PyTorch's AOT
  Inductor stable C ABI, BSD-3-Clause, carrying the upstream Facebook,
  Idiap, Deepmind, NEC and NYU copyright lines, but NOTICE listed only
  DLPack. A PyTorch entry is added with the full copyright block. The
  accompanying aoti_shim.def carries no copyright line of its own and is
  covered explicitly by that entry rather than given an NVIDIA header,
  since it declares the same upstream symbol names. The DLPack entry now
  also records where it is vendored.

LICENSE files (all five)
  Every LICENSE ended at "END OF TERMS AND CONDITIONS", omitting the
  required "APPENDIX: How to apply the Apache License to your work" and
  its boilerplate. Appended to all five. The text is verified identical
  to the canonical Apache 2.0 appendix.

Verified: 0 files with a non-conforming copyright string; check_spdx.py
passes over all 868 in-scope tracked files with the tightened regex.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Rob Parolin <rparolin@nvidia.com>

* docs: document per-subproject license files in root README

OSRB (NVBUG 4707569, comment NVIDIA#22) flagged the four sub-component LICENSE
files as redundant with the root LICENSE and asked for either their removal
or a root README Licensing section naming each subproject, its license and
its license path.

Each subproject builds an independent wheel and resolves its license file
relative to its own root, so the copies are kept and documented instead of
removed. Verified that the copies reach the built wheels: building
cuda_pathfinder produces dist-info/licenses/LICENSE even though its
pyproject.toml declares no explicit license-files (setuptools' default
LICEN[CS]E* glob covers it), as is also the case for cuda_core.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Signed-off-by: Rob Parolin <rparolin@nvidia.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…NVIDIA#2496)

* Migrate toolshed and ci helper scripts from os.path to pathlib

Part 7 of the series proposed in NVIDIA#2410.

Path joining and filesystem predicates in the toolshed and ci/tools helper
scripts now go through pathlib. glob.glob in dump_cutile_b64.py becomes
Path.glob, with the mtime key reading Path.stat().

Kept on os.path, with a comment where it is not obvious:

- os.path.abspath in build_static_bitcode_input.py, since sys.path wants a str
  and Path.absolute() does not normalize.
- os.path.isfile in check_generated_file_seals.py. That guard exists to skip
  anything that is not a readable regular file, and Path.is_file() is not a
  drop-in: it propagates OSError for errnos outside pathlib's ignore list
  (EACCES, ENAMETOOLONG) where os.path.isfile returns False.
- os.path.normpath in check_spdx.py, which already carries its own comment.

The plan on NVIDIA#2410 also listed a root conftest.py; there is no such file. The
three conftest.py files live under cuda_pathfinder, cuda_core and
cuda_bindings, and none of them use os.path.

Verified locally: ci/tools/tests/test_check_release_notes.py passes (42
tests), and check_spdx.py and check_generated_file_seals.py produce output
identical to the pre-change scripts when run over every tracked .py file.

Signed-off-by: LeSingh1 <sshaurya914@gmail.com>

* Return Path from notes_path; use Path.is_file in seal checker

Per review: treat these helper scripts as private, so notes_path can return
Path and drop the str/Path round-trip at its call site. Accept the behavioral
change from os.path.isfile to Path.is_file in check_generated_file_seals.

* Review: thread Path through check_release_notes, drop remaining os.path

Follow-up to mdboom's review.

- repo_root is now a Path end to end: load_backport_branch, check_release_notes
  and validate_backport_decision take Path, and --repo-root parses with
  type=Path. That removes the Path(repo_root) re-wrap inside the functions and
  the 19 str(tmp_path) conversions the tests needed to call them. The five
  main() argv lists keep str(): those are command-line strings, which argparse
  then turns back into a Path.
- build_static_bitcode_input: the last os.path use (os.path.abspath) becomes
  Path.resolve(); the os import is now unused and is dropped.

---------

Signed-off-by: LeSingh1 <sshaurya914@gmail.com>
Co-authored-by: Michael Droettboom <mdboom@gmail.com>
def supportsCudaAPI(name):
        return name in dir(cuda) or dir(cudart)

parses as `(name in dir(cuda)) or dir(cudart)`. `dir(cudart)` is a non-empty
list for any module, so it is unconditionally truthy and the function returns
a truthy value for every input, including names that exist nowhere.

The left operand is dead too: `cuda` is cuda.bindings.driver and every name
passed in is a cudaXxx runtime symbol. cudaGraphGetId, cudaGreenCtxCreate,
cudaDeviceGetExecutionCtx and cudaGraphConditionalHandleCreate are all defined
in runtime.pyx and appear nowhere in driver.pyx, so `name in dir(cuda)` is
always False and the result is always the `dir(cudart)` list.

Consequence: `not supportsCudaAPI(...)` is always False, so the API-presence
half of all 17 skipif guards that use it (lines 1443-1954) never fires. On a
build whose bindings genuinely lack the API, the test runs and dies with
AttributeError instead of skipping; only the driver_version_less_than() half
of each guard does any work.

Adds test_supportsCudaAPI, pinning all three cases: a runtime-only name, a
driver-only name, and a name that exists in neither. The last two fail before
this change.
* Catch up to current cybind main

* Bugfix for get_buffer_pointer
…VIDIA#2560)

def supports_nvlink(device):
        fields = nvml.FieldValue(1)
        fields[0].field_id = nvml.FI.DEV_NVLINK_GET_STATE

There is no `FI` attribute on cuda.bindings.nvml. The enum is `FieldId`
(nvml.pyx:1229), with DEV_NVLINK_GET_STATE at nvml.pyx:1454, and the sibling
test uses the correct spelling: test_nvlink.py:19 does
`fields[0].field_id = nvml.FieldId.DEV_NVLINK_LINK_COUNT`.

So the helper raises AttributeError on its first line of real work. Nobody
has noticed because it has no callers -- a repo-wide grep for
`supports_nvlink` finds only its own definition. Contrast util.supports_ecc,
which is called from test_page_retirement.py.

Adds tests/nvml/test_util.py, which stubs nvml.device_get_field_values so the
helper can be exercised without an NVLink-capable device, and asserts both
that it returns True and that it queried FieldId.DEV_NVLINK_GET_STATE. It
fails with AttributeError before this change.
…lue (NVIDIA#2535)

`cuda/core/__init__.py` reads `CUDA_CORE_DONT_FIX_TAB_COMPLETION` with a
bare `int(os.environ.get(..., "0"))` at import time. `int()` raises for any
value that is not a base-10 integer, and `os.environ.get` returns the empty
string (not the `"0"` default) when the variable is set but empty, so:

    export CUDA_CORE_DONT_FIX_TAB_COMPLETION=
    python -c "import cuda.core"
    ValueError: invalid literal for int() with base 10: ''

Clearing a variable with `export VAR=` is the usual way to neutralize it in
a shell profile, a Dockerfile, or a CI job spec, and `=true` / `=yes` are
the obvious guesses for a boolean-looking opt-out. All of them make the
whole package unimportable, which is a hard failure for a knob whose only
purpose is to skip an optional `rlcompleter` patch.

Parse the value leniently instead. Integer values keep their existing
meaning (non-zero opts out, so `0` and `00` still install the patch), while
a non-integer, non-empty value is honored as an opt-out rather than being
silently ignored. Unset and empty/whitespace-only both mean "not set".

Also document the variable, which was not listed on the environment
variables page, and drop the stale "only installed in interactive mode"
comment: the interactivity gate was intentionally removed in NVIDIA#2055 ("Always
install the monkeypatch"), so the patch has been unconditional since then.

The new parametrized test asserts the resulting behavior for eight values;
four of them ("", "   ", "true", "yes") fail on main because the subprocess
exits non-zero with the ValueError above.

Co-authored-by: Michael Droettboom <mdboom@gmail.com>
Docstrings across cuda_core still spelled parameter types with the
pre-3.10 typing generics. Replace Union[...] and Optional[...] with the
| form the rest of the package already uses, e.g. `stream : Stream |
None, optional` in _memoryview.pyx.

Docstrings only, so the .pyi changes are the stubgen-pyx output for the
edited .pyx files and no runtime behavior moves. In _module.pyx this also
realigns the max_potential_block_size docstring with its signature, which
already reads int | driver.CUoccupancyB2DSize.

Two code-level spellings stay as they are:

- LinkerHandleT in _linker.pyx is a runtime value, not an annotation.
  _program.pyx builds ProgramHandleT from it with `nvrtc.nvrtcProgram |
  int | LinkerHandleT`, and PEP 604 `|` on the forward-reference strings
  it holds raises TypeError.
- The union_type literal in _process_define_macro is error-message text
  rather than a docstring.

Sequence[...] and Iterable[...] elsewhere in cuda_core are
collections.abc generics and are unaffected.

Signed-off-by: Aryan <aryansputta@gmail.com>
Co-authored-by: Michael Droettboom <mdboom@gmail.com>
* Use subtests where appropriate everywhere

* Fix test

* Test every fan in a separate subtest

* Recognize skipped pytest subtests in CI logs

* Isolate independent inner test cases

* Narrow the cooler unsupported-call scope

* Contain fan-count failures per device

* Use stable identifiers for device subtests

* Fold nested subtest context managers

* Preserve the existing power-limit getter guard

* Guard memory affinity on pre-Kepler devices

* Keep invalid subtest results contained

---------

Co-authored-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com>
* Update PR guidance for organization-owned forks

* Clarify agent remote-write policy

* Add fork-aware pull request skill
…handle (NVIDIA#2551)

get_cuda_native_handle() wraps both the registry lookup and the getter call
in one try:

    try:
        return _handle_getters[obj_type](obj)
    except KeyError:
        raise TypeError("Unknown type: " + str(obj_type)) from None

The except clause is meant for "this type has no registered getter", but it
also fires for a KeyError raised *inside* the getter. When that happens the
diagnosis is wrong twice over: the reported type is registered, and
`from None` suppresses the context so the traceback that would show the real
failure is gone.

    >>> _add_cuda_native_handle_getter(Registered, getter_that_raises_keyerror)
    >>> get_cuda_native_handle(Registered())
    TypeError: Unknown type: <class 'Registered'>

Move the getter call out of the try. The unregistered-type path is
unchanged, which the existing test_get_handle_error still covers.
…IA#2563)

discover_benchmarks() goes out of its way to avoid def-time binding, and
says so:

    # Resolve the default inside the call so tests (and embedders) can
    # monkeypatch ``BENCH_DIR`` at the module level - Python binds default
    # args at def-time, so a literal default would ignore later patches.
    if bench_dir is None:
        bench_dir = BENCH_DIR

main() then reintroduces exactly that binding:

    def main(
        *,
        bench_dir: Path = BENCH_DIR,
        default_output: Path = DEFAULT_OUTPUT,
        ...
        registry = discover_benchmarks(bench_dir=bench_dir, ...)

Because main() always passes a non-None bench_dir down, the sentinel branch
in discover_benchmarks() can never be taken on this path, and patching
runner.main.BENCH_DIR - the documented mechanism - has no effect on main().
Same for DEFAULT_OUTPUT.

run_pyperf.py calls main() with no arguments, so this is the production
path. The existing tests patch BENCH_DIR and call discover_benchmarks()
directly, which is why the gap is invisible today.

Apply the same sentinel to both parameters. Explicit arguments keep working
unchanged, so the embedder API is unaffected.

Adds test_main_honors_a_monkeypatched_bench_dir, which patches BENCH_DIR to a
tmp dir holding one bench_*.py and drives main() with --list. It fails before
this change (main() lists the repo's real benchmarks instead).
* cuda.core: Add copy_batch to cuda.core.utils

* fallback for CUDA 12 and type annotations

* be more precise about CUDA requirements

* skip tests on Windows that require managed memory

* rework some tests

* Deduplicate _to_cumemlocation

* add missing file

* address review feedback

* review feedback: don't assume NUMA capabilities

* review feedback: clarify buffer requirements for async batched copies

* review feedback: explicitly reject special default streams

* review feedback: explicitly reject capturing streams

* review feedback: drop warning about unsupported PREFER_OVERLAP_WITH_COMPUTE hint

* review feedback: add missing descriptions for copy options values

* review feedback: align CopyOptions validation with existing practice

* review feedback: drop conditional imports for type checking

* account for CUDA 12/13 driver differences

* CUDA 12: drop rejection of unsupported copy options

* simplify tests
…or (NVIDIA#2463)

* fix(cuda.bindings): make cythonization warning-clean and enable -Werror

Clear the Cython warnings that blocked matching cuda.core's warning_errors
setting (NVIDIA#2450): drop ignored except clauses on Python-returning cudla
cpdefs, declare LOAD_LIBRARY_SEARCH_SYSTEM32 as const in windll.pxd, and
enable Cython Options.warning_errors in build_hooks.

Add source-level regression tests so these patterns do not return.

Signed-off-by: Omar Atie <atiaomar1978-hub@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>

* style: ruff-format cython warning cleanliness tests

Signed-off-by: Omar Atie <atiaomar1978-hub@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>

* test(cuda.bindings): drop cython warning cleanliness tests

Address review feedback: warning_errors in build_hooks already
guards against Cython warning regressions, so the source-level
tests add unnecessary maintenance cost.

Signed-off-by: Omar Atie <atiaomar1978-hub@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Signed-off-by: Omar Atie <atiaomar1978-hub@users.noreply.github.com>
Co-authored-by: Omar Atie <atiaomar1978-hub@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
…ed (NVIDIA#2539)

`check_generated_file_seals.py` declares three comment styles for the seal
line, one per generated-file family:

    _COMMENT_CHARS = {".py": b"#", ..., ".rst": b"..", ".c": b"//",
                      ".cpp": b"//", ".h": b"//"}

and `validate_generated_file_seal` compares the seal's captured prefix
against `expected_comment_prefix(filepath)` so a `.rst` file cannot be
sealed with a `#`, and so on. But the marker regex only ever accepts two of
the three:

    rb"^(?P<prefix>#|\.\.) "

`//` can never be captured, so `fullmatch` returns None for any sealed
`.c` / `.cpp` / `.h` file and it is rejected as `MALFORMED generated-file
seal` before the prefix comparison runs at all. The `b"//"` entries in
`_COMMENT_CHARS` and the branch that would validate them are dead.

Add `//` to the alternation, with a note tying it to `_COMMENT_CHARS` so
the two do not drift again.

This also adds the first tests for the script, under `toolshed/tests/`, and
runs them alongside the existing `ci/tools/tests` in the nightly tooling
job. The parametrized case is driven from `_COMMENT_CHARS` itself, so a
future entry whose prefix the regex cannot match fails immediately instead
of silently becoming dead code.
…VIDIA#2526)

* cuda.core: capture bound contexts for buffer deallocation streams

Record a DeallocationStream at device-pointer creation so default-stream tokens pin the allocation context (and PTDS the allocating thread) instead of relying on ambient state at free time.

* cuda.core: activate bound context during device-pointer teardown

Make the deallocation stream's context current around free/unmap/MR cleanup so destruction no longer depends on ambient CUDA context, and wire cuCtxSetCurrent into the resource-handles driver table.

* cuda.core: record from_handle deallocation streams at creation

Add keyword-only stream= on Buffer/ManagedBuffer.from_handle when mr owns the pointer, bind it at construction, and cover teardown with no or foreign current context.

* cuda.core: fail loudly on MemoryResource free errors

Stop treating CUDA_ERROR_INVALID_CONTEXT as a successful pool free, and let
explicit mr.deallocate() raise; destruction still contains errors in the
callback. Document PTDS deallocation ordering on the stream parameters and
note the context-safe Buffer teardown fix in the 1.2.0 release notes.

* cuda.core: reject incomplete buffer deallocation recipes

Require default deallocation streams to bind a current context at creation so teardown never relies on an ambiguous ambient token. Expand coverage and documentation for context-independent cleanup and failure reporting.

* cuda.core: initialize context when unpickling IPC buffers

Ensure spawned children can bind the imported buffer's default deallocation stream before their process target starts.

* test(cuda.core): set a current context in DLPack failure tests

Creating a Buffer with an owning memory resource now records a default
deallocation stream, which requires a current context. These two tests
never set one, so they passed or failed depending on whether the
preceding test left a context current under pytest-randomly.

* test(cuda.core): address review feedback on deallocation-stream PR

- Parametrize test_from_handle_mr_records_default_stream,
  test_from_handle_mr_records_explicit_stream, and
  test_from_handle_stream_requires_mr with [Buffer, ManagedBuffer] to
  cover the ManagedBuffer.from_handle entry point directly.
- Add test_close_with_default_stream_requires_context covering the
  _require_deallocation_stream_context guard in Buffer_close.
- Lift Stream_accept and default_stream to module-level imports.
- Replace _require_deallocation_stream_context (a pre-flight that
  duplicated make_deallocation_stream's context check) with
  _apply_deallocation_stream, which calls set_deallocation_stream once
  and translates CUDA_ERROR_INVALID_CONTEXT into a descriptive
  RuntimeError. Removes the redundant cuCtxGetCurrent call on the
  default-stream success path.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
…#2618)

* cuda.core: minor refactoring to prepare for copy with options

* inline capability check helper
* ci: add selective wheel test plumbing

* ci: update selective wheel test callers

* ci: enable nightly NumPy for metapackage tests

* ci: install exact local wheels in metapackage tests

* ci: simplify local wheel selection
…ont-update-2026-08-14

# Conflicts:
#	.github/workflows/build-wheel.yml
#	.github/workflows/test-sdist-linux.yml
#	.github/workflows/test-sdist-windows.yml
#	cuda_bindings/cuda/bindings/_internal/cudla.pxd
#	cuda_bindings/cuda/bindings/_internal/cudla_linux.pyx
#	cuda_bindings/cuda/bindings/_internal/cudla_windows.pyx
#	cuda_bindings/cuda/bindings/_internal/driver_linux.pyx
#	cuda_bindings/cuda/bindings/_internal/driver_windows.pyx
#	cuda_bindings/cuda/bindings/_internal/nvfatbin_linux.pyx
#	cuda_bindings/cuda/bindings/_internal/nvfatbin_windows.pyx
#	cuda_bindings/cuda/bindings/_internal/nvjitlink.pxd
#	cuda_bindings/cuda/bindings/_internal/nvjitlink_linux.pyx
#	cuda_bindings/cuda/bindings/_internal/nvjitlink_windows.pyx
#	cuda_bindings/cuda/bindings/_internal/nvml_linux.pyx
#	cuda_bindings/cuda/bindings/_internal/nvml_windows.pyx
#	cuda_bindings/cuda/bindings/_internal/nvrtc_linux.pyx
#	cuda_bindings/cuda/bindings/_internal/nvrtc_windows.pyx
#	cuda_bindings/cuda/bindings/_internal/nvvm_linux.pyx
#	cuda_bindings/cuda/bindings/_internal/nvvm_windows.pyx
#	cuda_bindings/cuda/bindings/_v2/nvrtc.pxd
#	cuda_bindings/cuda/bindings/_v2/nvrtc.pyx
#	cuda_bindings/cuda/bindings/cudla.pxd
#	cuda_bindings/cuda/bindings/cudla.pyx
#	cuda_bindings/cuda/bindings/cufile.pyx
#	cuda_bindings/cuda/bindings/cycudla.pxd
#	cuda_bindings/cuda/bindings/cycudla.pyx
#	cuda_bindings/cuda/bindings/cydriver.pxd
#	cuda_bindings/cuda/bindings/cynvfatbin.pxd
#	cuda_bindings/cuda/bindings/cynvjitlink.pxd
#	cuda_bindings/cuda/bindings/cynvjitlink.pyx
#	cuda_bindings/cuda/bindings/cynvml.pxd
#	cuda_bindings/cuda/bindings/cynvrtc.pxd
#	cuda_bindings/cuda/bindings/nvfatbin.pxd
#	cuda_bindings/cuda/bindings/nvfatbin.pyx
#	cuda_bindings/cuda/bindings/nvjitlink.pxd
#	cuda_bindings/cuda/bindings/nvjitlink.pyx
#	cuda_bindings/cuda/bindings/nvml.pxd
#	cuda_bindings/cuda/bindings/nvml.pyx
#	cuda_bindings/cuda/bindings/nvrtc.pyx
#	cuda_bindings/cuda/bindings/nvvm.pxd
#	cuda_bindings/cuda/bindings/nvvm.pyx
#	cuda_bindings/docs/source/module/driver.rst
#	cuda_bindings/docs/source/module/nvrtc.rst
#	cuda_bindings/docs/source/module/runtime.rst
#	cuda_bindings/tests/nvml/test_device.py
#	cuda_core/tests/system/test_system_device.py
@copy-pr-bot

copy-pr-bot Bot commented Aug 15, 2026

Copy link
Copy Markdown
Contributor

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@rwgk rwgk added this to the cuda.bindings 13.4.0 & 12.9.8 milestone Aug 15, 2026
@rwgk rwgk added enhancement Any code-related improvements P0 High priority - Must do! CI/CD CI/CD infrastructure cuda.bindings Everything related to the cuda.bindings module cuda.core Everything related to the cuda.core module labels Aug 15, 2026
@github-actions github-actions Bot added the CI/CD CI/CD infrastructure label Aug 15, 2026
@rwgk rwgk added the cuda.pathfinder Everything related to the cuda.pathfinder module label Aug 15, 2026
@github-actions github-actions Bot added cuda.bindings Everything related to the cuda.bindings module cuda.core Everything related to the cuda.core module cuda.pathfinder Everything related to the cuda.pathfinder module labels Aug 15, 2026
@rwgk rwgk self-assigned this Aug 15, 2026
@rwgk

rwgk commented Aug 15, 2026

Copy link
Copy Markdown
Contributor Author

codex gpt-5.6-sol ultra:

Noteworthy merge-main nuances

I want to record the provenance of d31a58c and 5830ba7, because the two fixes appeared one after the other during testing but address independent problems. This also gives us concrete expectations for the next ctk-next merge/transfer-preview cross-check.

Why the cuDLA and CUmemLocation failures appeared sequentially

The merge commit a6ca4f brought the later public main into the branch from PR #2588. Public main included the current-generator update from #2603 (ffb776c) as well as the change that treats Cython warnings as errors (8457bb0). The generated bindings conflicted because the two sides represented different toolkit surfaces: the release branch had CUDA 13.4 output, whereas public main still had CUDA 13.3 output. Keeping the 13.4 files was necessary, but keeping those files wholesale also accidentally discarded the toolkit-independent generator fixes from #2603.

The first visible result was the Linux build failure seen while testing #2623. The stale 13.4 cuDLA output still put except * on the Python-object-returning device_get_attribute and module_get_attributes wrappers. Current Cython reports that exception clause as invalid/ignored, and the project's warnings-as-errors setting makes the diagnostic fatal. Every affected job therefore stopped while building cuda.bindings.

Commit d31a58c repairs that merge accident in the release-appropriate way: it regenerates from the CUDA 13.4 inputs using the current cybind generator. It does not copy CUDA 13.3 generated files from main. The result retains the CUDA 13.4 API surface while recovering the generic generator fixes from #2603, including removal of the invalid cuDLA exception clauses.

Only after that repair could the build move on to cuda.core. The wheel workflow builds and installs cuda.bindings before it Cythonizes cuda.core; the earlier cuDLA error prevented the latter phase from running at all. The next local build therefore became the first build in this sequence to reach the CUmemLocation sites, where Cython reported:

Not all members given for struct 'CUmemLocation'

This is what I consider a latent public 13.4.x problem. The incompatible declaration and initialization syntax were already together in the PR #2588 base (9d4142), before either d31a58c or 5830ba7. PR #2588 did not exercise this full compilation path; its available checks were metadata/label checks, while the relevant build coverage was not run. Regenerating the bindings did not add the localized member and did not cause the cuda.core problem. It merely removed the earlier blocker and allowed the sequential build to reveal it.

Where 5830ba7 comes from

Public main's CUDA 13.3-generated declaration of CUmemLocation had two visible members, type and id. In that layout, the aggregate initializer introduced by #2434 was complete:

cydriver.CUmemLocation(type=..., id=...)

CUDA 13.4 adds a localized arm to the anonymous location union. The generated Cython declaration consequently exposes type, id, and localized. Cython 3.2 requires an aggregate initializer to name every member in its declaration, so all of the two-member aggregate initializers fail against the 13.4 declaration.

The correct source-compatible form is to declare the struct and assign only the active fields directly:

cdef cydriver.CUmemLocation location
location.type = ...
location.id = ...

All location kinds currently constructed by cuda.core use the id arm (DEVICE, HOST, HOST_NUMA, or HOST_NUMA_CURRENT). The localized arm belongs to DEVICE_LOCALITY_DOMAIN, which these paths do not construct. Direct field assignment is therefore both semantically correct for CUDA 13.4 and compatible with the older two-member generated declaration.

This was first repaired on the ctk-next side by ctk-next commit cf9491a54ed ("Avoid aggregate CUmemLocation initialization"). Public commit #2593 (29acb74) subsequently refactored the shared conversion into the new _memory/_location.pxd. Commit 5830ba7 is the same ctk-next fix adapted to that newer public tree: it preserves the original five behavioral repairs and also updates the new shared helper. That is why 5830ba7 touches six files while the original ctk-next change touched five.

Why ctk-next being in sync did not already put this fix in public

The direction of the two companion ctk-next PRs matters:

  • ctk-next PR 518 was the authoritative merge view. It brought the public history represented by PR Merge main into 13.4.x #2588 into ctk-next while retaining ctk-next-only work, including ctk-next commit cf9491a54ed.
  • ctk-next PR 519 was the companion public-shaped transfer preview, based exactly on PR Merge main into 13.4.x #2588 at 9d41426. Its non-generated transfer commit was 7ca3fb5a855; five of its ten changed files carried the original CUmemLocation repair.

PR 519 was a review artifact and was closed-as-planned rather than merged into public. Thus ctk-next contained the public PR #2588 history, but the ctk-next-only correction had only been previewed in the reverse direction. It had not reached the public 13.4.x branch. PR #2641 now carries the adapted public-tree version in 5830ba7.

Cross-check for the next ctk-next merge and preview

After merging the exact PR #2641 head into ctk-next and regenerating the CUDA 13.4.1 outputs, I plan to create both the authoritative merge PR and its public-shaped transfer preview. The old PR 519 transfer commit 7ca3fb5a855 gives us a useful independent reference.

For the new non-generated transfer commit, the expected comparison is:

  • The five old CUmemLocation hunks, and the newer _memory/_location.pxd hunk, should not appear. They are now already present in the public preview base through 5830ba7.
  • The other five substantive ctk-next-only changes from 7ca3fb5a855 should remain unless public has independently acquired them: cuda_bindings/tests/cython/build_tests.bat, cuda_bindings/tests/cython/build_tests.py, cuda_core/tests/example_tests/test_basic_examples.py, cuda_core/tests/system/test_system_system.py, and cuda_core/tests/test_green_context.py.
  • Generated bindings and generated stubs should be compared separately. The public branch uses CUDA 13.4.0 output, whereas the new ctk-next merge and preview will regenerate based on a newer CUDA 13.4. The strong check there is byte-for-byte agreement between the new merge view and the new preview view, not identity with either old PR 519 or public d31a58c.
  • The comparison should be semantic/path-by-path rather than expecting the entire new transfer commit to have the same patch ID as 7ca3fb5a855, because public cuda.core: Add copy_batch to cuda.core.utils #2593 changed the source topology and later public work changes surrounding context.

If a CUmemLocation change reappears in the new transfer commit, that would indicate either a stale preview base or a merge resolution that lost 5830ba7. If one of the unrelated five ctk-next-only changes disappears without an equivalent public change, that would indicate that the merge lost ctk-next work. Unexplained new non-generated files would similarly point to a base, conflict-resolution, or transfer-attribution problem.

With d31a58c, 5830ba7, and the regenerated stubs applied, the full local CUDA 13.4 build and test runbooks pass: pathfinder, bindings in both default-stream modes, bindings Cython tests, cuda.core, and cuda.core Cython tests. This confirms that the two commits remove successive blockers without replacing or narrowing the CUDA 13.4 API surface.

@rwgk

rwgk commented Aug 15, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test

@copy-pr-bot

copy-pr-bot Bot commented Aug 15, 2026

Copy link
Copy Markdown
Contributor

/ok to test

@rwgk, there was an error processing your request: E1

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/1/

@rwgk

rwgk commented Aug 15, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 24fd7b0

rwgk added 2 commits August 14, 2026 23:13
Sort completed branch runs explicitly and choose the newest successful run whose required artifacts are still available. Validate bindings and metapackage artifacts at all prior-branch download sites, and cover fallback and compatibility behavior with standalone tests.
@rwgk rwgk closed this Aug 15, 2026
@rwgk
rwgk deleted the 13.4.x-merge-main-cont branch August 15, 2026 06:37
@rwgk rwgk reopened this Aug 15, 2026
@rwgk

rwgk commented Aug 15, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 1ebebc5

@rwgk

rwgk commented Aug 15, 2026

Copy link
Copy Markdown
Contributor Author

I added two follow-up commits based on the failures in CI run 31865723802.

Restore compatibility coverage for clock event reasons (df5356d)

All 11 failing Linux AArch64 test jobs failed at the same enum-coverage case. cuda.core exposes ClocksEventReasons.BOARD_LIMIT and ClocksEventReasons.RELIABILITY, but the older cuda.bindings versions used by the CUDA 12.9 and 13.3 compatibility rows do not yet expose the corresponding NVML enum names.

The production mapping already handles this intentionally: when those two names are absent, it uses their stable numeric NVML values. The generic coverage test had lost the matching compatibility allowance, so it incorrectly required an older binding to contain names that only a newer binding provides.

This commit restores the narrow, dynamic test guard from ctk-next commit 91aa75d:

  • each wrapper member is exempted only when its corresponding binding member is absent;
  • with CUDA 13.4 bindings, both names are present, the exemption set is empty, and the normal full-coverage assertion still applies;
  • no runtime behavior and no other enum coverage is changed.

This is therefore a compatibility-test correction rather than a blanket skip. It lets the intended older-bindings matrix continue checking all members that those bindings can actually provide.

Harden prior-branch artifact lookup (1ebebc5)

Two Linux wheel-build jobs failed after their current CUDA 13.4 builds had succeeded, while assembling the CUDA 12 half of the combined cuda.core wheel. Each matrix job independently asked gh run list for one successful run on 12.9.x and assumed that result was the newest usable run.

That assumption did not hold in this CI run: jobs started within roughly the same minute resolved several different historical run IDs. The Python 3.14t job selected a run whose matching artifact had expired, while the Python 3.15 job selected a run from before that artifact existed. The logs do not establish which GitHub/CLI layer produced the stale ordering, so this change does not depend on attributing the behavior to a specific external bug.

The lookup now:

  • fetches a bounded set of completed branch runs, filters successful runs explicitly, and sorts them by creation time;
  • scans newest to oldest and uses the paginated Actions artifact API to require a matching, unexpired bindings wheel before selecting a run;
  • also requires cuda-python-wheel when a test job will download it, so all requested artifacts must come from the same run;
  • distinguishes the bindings wheel artifact from its -tests sibling, preventing a tests-only match from qualifying a run;
  • preserves branch lookup without artifact requirements, including the nightly run_id + head_sha output, and leaves tag lookup unchanged.

All three prior-branch consumers now use the validated lookup: the wheel builder and the Linux and Windows wheel-test workflows.

The new standalone tests use a fake gh executable to cover unordered and failed runs, a workflow selected by filename rather than display name, fallback past expired/missing artifacts, multiple artifacts required from one run, a live -tests artifact with an expired wheel, and artifact-API failure propagation.

Validation

  • cuda_core/tests/test_enum_coverage.py: 30 passed with --noconftest
  • ci/tools/tests: 60 passed with --noconftest
  • bash -n and standalone shellcheck for ci/tools/lookup-run-id
  • actionlint, YAML, Ruff, SPDX, and git diff --check
  • full pre-commit run --all-files passed with check-pixi-cuda-version intentionally skipped because CUDA 13.4 Conda packages are not available yet

The CI rerun remains the authoritative cross-version check: the local environment has CUDA 13.4 bindings, whereas the previously failing rows exercise the conditional enum guard with CUDA 12.9/13.3 bindings and the real prior-branch artifact inventory.

@rwgk rwgk mentioned this pull request Aug 15, 2026
2 tasks
@rwgk

rwgk commented Aug 15, 2026

Copy link
Copy Markdown
Contributor Author

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI/CD CI/CD infrastructure cuda.bindings Everything related to the cuda.bindings module cuda.core Everything related to the cuda.core module cuda.pathfinder Everything related to the cuda.pathfinder module enhancement Any code-related improvements P0 High priority - Must do!

Projects

None yet

Development

Successfully merging this pull request may close these issues.