Skip to content

GPU device loss fails loudly in one test file and silently in the other #883

Description

@ocots

Summary

test/helpers/capabilities.jl implements the Handbook's capability-gated test policy
(philosophy/testing.md, §"Capability-gated tests"), including the loud-failure guard
on_gpu_runner(). But the guard is wired into one of the two GPU skip sites. On the kkt
runner, a lost device fails loudly in one file and skips silently in the other.

The policy

Handbook/philosophy/testing.md:122-133:

On the runner that is supposed to have the capability, make its absence fail loudly
instead of silently.

on_gpu_runner() = get(ENV, "RUNNER_NAME", "") in ("kkt", "occidata")

if on_gpu_runner()
    Test.@test is_cuda_on()   # fails loudly if the GPU runner lost its device
end

Reference: CTSolvers.jl#189 / #190, CTFlows.jl#375.

What we have

Both device-tier tests correctly use Test.@test_skip rather than a silent early return, so
they show as Broken in the summary rather than vanishing. That half is right, and both carry
a comment explaining the choice.

The loud guard is not symmetric:

site @test_skip on_gpu_runner() guard
test/suite/flows/test_gpu_routing.jl:197-204 yes yes
test/suite/builders/test_options_forwarding.jl:66-70 yes no

test_options_forwarding.jl does not import on_gpu_runner at all — its using .TestCapabilities line takes only is_cuda_on.

So on a kkt whose driver broke, test_gpu_routing.jl fails as designed, but
backend (CUDA) in test_options_forwarding.jl reports Broken — indistinguishable from a
laptop run. That is a smaller version of the failure class the pattern exists to prevent.

Also worth a look

  • on_gpu_runner() matches the single literal "kkt", while the Handbook snippet now reads
    in ("kkt", "occidata"). For this repository "kkt" is still correct — .github/workflows/CI.yml:39
    is runs_on: '[["kkt"]]' and there is no occidata job. But CTParser 0.9.0 has already
    retired kkt for occidata on its side. If OptimalControl follows, this literal must move
    with it — and its own docstring notes it "stops firing silently rather than failing loudly"
    if the runner is renamed.
  • gpu_extension_armed() is defined and documented but never asserted on the GPU runner. On
    kkt the extension being unarmed (the MadNLPGPU + CUDA + CUDSS trigger, CTSolvers#189)
    is as much a broken-runner condition as a missing device.

Proposed fix

  1. Import on_gpu_runner in test_options_forwarding.jl and add the same
    if on_gpu_runner(); Test.@test is_cuda_on(); end guard.
  2. Consider asserting gpu_extension_armed() under the same guard.
  3. Keep the runner literal in capabilities.jl and CI.yml in sync; a comment on each
    pointing at the other would make the coupling visible.

Found while raising the compat bounds (CTBase 0.29 / CTModels 0.18 / CTFlows 0.17 /
CTParser 0.9 / CTDirect 1.1 / ExaModels 0.12); the full suite is green, 2242 pass / 0 fail,
with these two as the only Broken.

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingunit testsImprovements or additions to the tests

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions