Local failures in test_green_context.py tests

[mdboom]

Is there something I need to do to get these to pass?

FAILED cuda_core/tests/test_green_context.py::TestSMResourceSplit::test_two_groups - cuda.core._utils.cuda_utils.CUDAError: CUDA_ERROR_INVALID_RESOURCE_CONFIGURATION: This error indicates one or more resources are insufficient or non-applicable ...
FAILED cuda_core/tests/test_green_context.py::TestGreenContextKernelLaunch::test_two_green_contexts_independent - cuda.core._utils.cuda_utils.CUDAError: CUDA_ERROR_INVALID_RESOURCE_CONFIGURATION: This error indicates one or more resources are insufficient or non-applicable ...
FAILED cuda_core/tests/test_green_context.py::TestContextResources::test_green_ctx_resources_reflect_partition - cuda.core._utils.cuda_utils.CUDAError: CUDA_ERROR_INVALID_RESOURCE_CONFIGURATION: This error indicates one or more resources are insufficient or non-applicable ...

________________________________________________________________ TestSMResourceSplit.test_two_groups ________________________________________________________________
self = <test_green_context.TestSMResourceSplit object at 0x71f99d592060>, sm_resource = <cuda.core._device_resources.SMResource object at 0x71f9786ad7c0>

    def test_two_groups(self, sm_resource):
        """Two-group split with explicit aligned counts."""                                                                                                                  half = _aligned_half(sm_resource)
        if half < sm_resource.min_partition_size:
            pytest.skip("Not enough SMs for a 2-group split")

>       groups, rem = sm_resource.split(SMResourceOptions(count=(half, half)))
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

half       = 24
self       = <test_green_context.TestSMResourceSplit object at 0x71f99d592060>
sm_resource = <cuda.core._device_resources.SMResource object at 0x71f9786ad7c0>

cuda_core/tests/test_green_context.py:246:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
cuda/core/_device_resources.pyx:475: in cuda.core._device_resources.SMResource.split
    ???                                                                                                                                                              cuda/core/_device_resources.pyx:286: in cuda.core._device_resources._split_with_general_api
    ???                                                                                                                                                              cuda/core/_utils/cuda_utils.pyx:65: in cuda.core._utils.cuda_utils.HANDLE_RETURN
    ???                                                                                                                                                              _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

>   ???
E   cuda.core._utils.cuda_utils.CUDAError: CUDA_ERROR_INVALID_RESOURCE_CONFIGURATION: This error indicates one or more resources are insufficient or non-applicable f
or the operation.


cuda/core/_utils/cuda_utils.pyx:163: CUDAError

[leofang]

Lowering this to P1 as hitting CUDA_ERROR_INVALID_RESOURCE_CONFIGURATION is actually expected by the driver design. Raised #2031 to improve the situation. Will add my understanding here before closing the issue.

[leofang]

Understanding SM split constraints and the backfill option

When using SMResource.split() to partition SM resources for green contexts, users may encounter CUDA_ERROR_INVALID_RESOURCE_CONFIGURATION even when the requested SM count is properly aligned to coscheduled_alignment. This is expected behavior — the CUDA driver's structured split API (cuDevSmResourceSplit) enforces hardware topology constraints beyond simple alignment.

What we found

• The driver has internal constraints on how co-scheduled SM groups can be formed. Not all partition sizes are achievable on every GPU, even when the count is a multiple of coscheduled_alignment.
• The set of valid partition sizes varies by GPU model and SKU. For example, on some desktop Blackwell GPUs (RTX 5070/5080), only a small subset of aligned counts produces a valid 2-group split.
• Discovery mode can help determine what the driver considers a valid partition:

  sm = dev.resources.sm
  groups, rem = sm.split(SMResourceOptions(count=None), dry_run=True)
  print(groups[0].sm_count)  # driver-chosen valid count

The fix — backfill=True

SMResourceOptions now supports a backfill field. When set to True, the driver relaxes the co-scheduling constraint and allows arbitrary aligned SM counts:

sm = dev.resources.sm
half = (sm.sm_count // 2 // sm.coscheduled_alignment) * sm.coscheduled_alignment

# Without backfill: may fail on some GPU topologies
groups, rem = sm.split(SMResourceOptions(count=(half, half)))

# With backfill: works on all topologies
groups, rem = sm.split(SMResourceOptions(count=(half, half), backfill=True))

The trade-off: with backfill, some SMs in each group may not have the co-scheduling guarantee (relevant for workloads using thread block clusters). For workloads that don't rely on clusters, backfill=True is the recommended option for maximum flexibility.

Alternatively, setting coscheduled_sm_count=2 removes the cluster co-scheduling requirement entirely, making every aligned-to-2 count valid without backfill.

See the CUDA driver API documentation for details on CU_DEV_SM_RESOURCE_GROUP_BACKFILL and the structured split API.

Fix: #2031

-- Leo's bot