Summary
test/helpers/capabilities.jl implements the Handbook's capability-gated test policy
(philosophy/testing.md, §"Capability-gated tests"), including the loud-failure guard
on_gpu_runner(). But the guard is wired into one of the two GPU skip sites. On the kkt
runner, a lost device fails loudly in one file and skips silently in the other.
The policy
Handbook/philosophy/testing.md:122-133:
On the runner that is supposed to have the capability, make its absence fail loudly
instead of silently.
on_gpu_runner() = get(ENV, "RUNNER_NAME", "") in ("kkt", "occidata")
if on_gpu_runner()
Test.@test is_cuda_on() # fails loudly if the GPU runner lost its device
end
Reference: CTSolvers.jl#189 / #190, CTFlows.jl#375.
What we have
Both device-tier tests correctly use Test.@test_skip rather than a silent early return, so
they show as Broken in the summary rather than vanishing. That half is right, and both carry
a comment explaining the choice.
The loud guard is not symmetric:
| site |
@test_skip |
on_gpu_runner() guard |
test/suite/flows/test_gpu_routing.jl:197-204 |
yes |
yes |
test/suite/builders/test_options_forwarding.jl:66-70 |
yes |
no |
test_options_forwarding.jl does not import on_gpu_runner at all — its using .TestCapabilities line takes only is_cuda_on.
So on a kkt whose driver broke, test_gpu_routing.jl fails as designed, but
backend (CUDA) in test_options_forwarding.jl reports Broken — indistinguishable from a
laptop run. That is a smaller version of the failure class the pattern exists to prevent.
Also worth a look
on_gpu_runner() matches the single literal "kkt", while the Handbook snippet now reads
in ("kkt", "occidata"). For this repository "kkt" is still correct — .github/workflows/CI.yml:39
is runs_on: '[["kkt"]]' and there is no occidata job. But CTParser 0.9.0 has already
retired kkt for occidata on its side. If OptimalControl follows, this literal must move
with it — and its own docstring notes it "stops firing silently rather than failing loudly"
if the runner is renamed.
gpu_extension_armed() is defined and documented but never asserted on the GPU runner. On
kkt the extension being unarmed (the MadNLPGPU + CUDA + CUDSS trigger, CTSolvers#189)
is as much a broken-runner condition as a missing device.
Proposed fix
- Import
on_gpu_runner in test_options_forwarding.jl and add the same
if on_gpu_runner(); Test.@test is_cuda_on(); end guard.
- Consider asserting
gpu_extension_armed() under the same guard.
- Keep the runner literal in
capabilities.jl and CI.yml in sync; a comment on each
pointing at the other would make the coupling visible.
Found while raising the compat bounds (CTBase 0.29 / CTModels 0.18 / CTFlows 0.17 /
CTParser 0.9 / CTDirect 1.1 / ExaModels 0.12); the full suite is green, 2242 pass / 0 fail,
with these two as the only Broken.
Summary
test/helpers/capabilities.jlimplements the Handbook's capability-gated test policy(
philosophy/testing.md, §"Capability-gated tests"), including the loud-failure guardon_gpu_runner(). But the guard is wired into one of the two GPU skip sites. On thekktrunner, a lost device fails loudly in one file and skips silently in the other.
The policy
Handbook/philosophy/testing.md:122-133:Reference: CTSolvers.jl#189 / #190, CTFlows.jl#375.
What we have
Both device-tier tests correctly use
Test.@test_skiprather than a silent early return, sothey show as
Brokenin the summary rather than vanishing. That half is right, and both carrya comment explaining the choice.
The loud guard is not symmetric:
@test_skipon_gpu_runner()guardtest/suite/flows/test_gpu_routing.jl:197-204test/suite/builders/test_options_forwarding.jl:66-70test_options_forwarding.jldoes not importon_gpu_runnerat all — itsusing .TestCapabilitiesline takes onlyis_cuda_on.So on a
kktwhose driver broke,test_gpu_routing.jlfails as designed, butbackend (CUDA)intest_options_forwarding.jlreportsBroken— indistinguishable from alaptop run. That is a smaller version of the failure class the pattern exists to prevent.
Also worth a look
on_gpu_runner()matches the single literal"kkt", while the Handbook snippet now readsin ("kkt", "occidata"). For this repository"kkt"is still correct —.github/workflows/CI.yml:39is
runs_on: '[["kkt"]]'and there is nooccidatajob. But CTParser 0.9.0 has alreadyretired
kktforoccidataon its side. If OptimalControl follows, this literal must movewith it — and its own docstring notes it "stops firing silently rather than failing loudly"
if the runner is renamed.
gpu_extension_armed()is defined and documented but never asserted on the GPU runner. Onkktthe extension being unarmed (theMadNLPGPU+CUDA+CUDSStrigger, CTSolvers#189)is as much a broken-runner condition as a missing device.
Proposed fix
on_gpu_runnerintest_options_forwarding.jland add the sameif on_gpu_runner(); Test.@test is_cuda_on(); endguard.gpu_extension_armed()under the same guard.capabilities.jlandCI.ymlin sync; a comment on eachpointing at the other would make the coupling visible.
Found while raising the compat bounds (CTBase 0.29 / CTModels 0.18 / CTFlows 0.17 /
CTParser 0.9 / CTDirect 1.1 / ExaModels 0.12); the full suite is green,
2242 pass / 0 fail,with these two as the only
Broken.