fix(gpu): cap the GPU test targets, count the aux retries, split the downgrade counter

[MauroToscano]

Three more follow-ups on the device-only downgrade recovery, continuing #921
(merged) on the same base. #921 hardened the recovery path by reading; these
come from watching it run.

Addendum (post-GPU-investigation)

The hang was observed live on rented 5090s: a panic on a device-only cliff
assert does not reliably abort the process — the panicking thread unwinds while
its siblings stay parked in CUDA driver waits, and the prover hangs forever —
so the merge-queue GPU targets now carry per-target timeouts rather than
holding the rented box until the workflow-level timeout kills the job with no
indication of which group stalled. The resident-aux retry counter makes decline
frequency observable in production, where the drain-and-retry usually succeeds
and, until now, left no trace but an eprintln nobody reads. And the downgrade
counter is split by site, because the investigation showed the two sites firing
independently: a preprocessed BITWISE table took the R1 resident-aux downgrade
despite never having been device-only, which the combined counter reported as a
device-only gate miss with nothing to tell the two apart. Each counter now names
its own fix — mirror the missing condition into the gate for R2, relieve VRAM
pressure for R1.

1. Per-target wall clock on the GPU test targets

scripts/gpu_test.sh runs five Makefile targets on the rented box with no
per-target limit, so a hung prover burns the box until the workflow timeout and
the log never says which group stalled. The four targets that run the prover
(test-cuda-integration, test-cuda-fallback, test-prover-cuda,
test-prover-comprehensive-cuda) are now wrapped in a shared
GPU_TEST_TIMEOUT := timeout -k 30 2700. 45 minutes is well above their normal
runtime and well below the job timeout; timeout's 124 exit fails the target,
so gpu_test.sh names the group and blocks the merge as it would for any other
failure. test-math-cuda is deliberately left alone — kernel parity never
enters the prover, so it has no exposure to this hang.

2. GPU_RESIDENT_AUX_RETRIES

The R1 resident-aux path retries the device LDE after a full device drain, and
that retry usually succeeds — which is exactly why it was invisible. The counter
is bumped where the "declined … retrying after device drain" eprintln fires, so
it counts declines rather than outcomes, and it is included in
reset_all_gpu_call_counters(). Read against the downgrade counter it separates
cause from consequence: retries with no downgrades means the drain absorbed the
pressure, both rising together means it no longer does.

3. Splitting GPU_DEVICE_ONLY_DOWNGRADES

The counter was incremented from two sites with different meanings:
materialize_lde_trace_host (R2), which is always a device-only gate miss, and
materialize_aux_trace_host (R1), which is entered whenever aux_resident() is
set — whatever the gate said — and so fires on tables that were never
device-only. Conflating them makes the number unusable for the thing it exists
to surface. GPU_DEVICE_ONLY_DOWNGRADES now counts the R2 site only, and the
new GPU_RESIDENT_AUX_DOWNGRADES counts R1; both doc comments state their sole
increment site, and the device_only_gate docs (rewritten by #920, which
described the combined semantics) now say which counter each round bumps and
that the gate does not govern the R1 one.

gpu_device_only_residency_fires_and_verifies asserted the old combined counter
was zero; it now asserts both are zero with per-site messages, the existing
gate-mirroring text staying on the device-only one.

Testing

make fmt clean, make lint green across all four passes including
--features lambda-vm-prover/cuda via the empty-cubin stubs, and
cargo check -p stark for the non-cuda build.

The touched code was not executed. No CUDA toolchain on this machine, and
the counters sit on the recovery path, which runs only when a device dispatch
declines under real VRAM pressure. The Makefile change is verified by
make -n expansion; the rest is by reading plus the merge-queue GPU job. Note
that the timeout wrapper assumes GNU coreutils timeout, which the Linux GPU
box has but stock macOS does not — these targets already require an NVIDIA GPU
and nvcc, so they were never runnable on a mac.