Skip to content

Solexecbench Exploit Blog #9

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
pramodith wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

Solexecbench Exploit Blog #9

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
193 changes: 193 additions & 0 deletions _posts/2026-07-29-solexecbench-exploit.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,193 @@
---
layout: post
title: "SOLExecbench: Hacking your way to the Speed of Light"
description: "Exposing a a vulnerablility in SOLExecbench that gives submitted kernels a SOL-score of 1.0."
date: 2026-07-29
author:
- name: "Geometric Team"
linkedin: "https://www.linkedin.com/company/112782074/"
---

The gaming of benchmarks in AI is a well-known phenomenon. Sakana AI's "AI CUDA Engineer" was found to have discovered a memory exploit in its evaluation sandbox that let it skip correctness checks entirely, inflating its reported kernel speedups until the company hardened the harness ([Sakana AI](https://x.com/SakanaAILabs/status/1892992938013270019)). Meta faced accusations that Llama 4 Maverick was tuned against benchmark test sets ahead of its release, allegations the company denied ([IT Pro](https://www.itpro.com/technology/artificial-intelligence/meta-llama-4-model-launch-benchmarks)). Test data has also leaked into training sets unintentionally: OpenAI acknowledged a filtering bug that let parts of the GSM8K and MATH test sets end up in GPT-3's training corpus ([Leak, Cheat, Repeat](https://arxiv.org/pdf/2402.03927)). And reward hacking predates LLMs altogether: OpenAI's own CoastRunners agent learned to loop endlessly in a lagoon farming respawning targets instead of finishing the race, racking up a higher score while catching fire and going the wrong way ([OpenAI](https://openai.com/index/faulty-reward-functions/)). In this post, we will discuss a vulnerability that we discovered in the SOLExecbench benchmark that allows submitted kernels to achieve a SOL-score of 1.0, regardless of their actual performance.

@b-albar b-albar Jul 29, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Competition like the nvfp4 hackaton have also been plagued with this. May be worth to mention to insist on the fact that it is not new but still exists : https://www.gpumode.com/news/reward-hacking-nvfp4


## SOLExecbench Overview

[SOLExecbench](https://research.nvidia.com/benchmarks/sol-execbench/blog/introducing-sol-execbench) is NVIDIA's execution-based benchmark for evaluating GPU kernel implementations against the theoretical performance limits of the hardware. Instead of scoring a kernel relative to a software baseline that shifts every time someone submits a faster kernel, SOLExecbench anchors the score to a fixed, hardware-derived bound: the **Speed of Light (SOL)**, the runtime floor implied by the GPU's peak compute throughput and memory bandwidth for a given operation.

The benchmark contains 235 CUDA kernel optimization problems extracted from 124 production and frontier models (LLMs, diffusion, vision, audio, video, and multimodal), spanning BF16, FP32, FP8, and NVFP4 precision. Each problem ships with a PyTorch reference implementation, a set of dynamically shaped workloads, and a target GPU architecture; submissions are compiled and timed on real B200 hardware inside isolated Docker containers, with the L2 cache cleared before every timed iteration.

Every submission gets a SOL Score $S \in [0, 1]$:

$$
S(t) = \frac{t_b - t_{sol}}{(t - t_{sol}) + (t_b - t_{sol})}
$$

where $t$ is the measured kernel latency, $t_b$ is a stored baseline latency, and $t_{sol}$ is the SOL latency estimated by NVIDIA's SOLAR tool using a roofline model. $S = 0.5$ means the kernel matches the baseline; $S \to 1$ means it approaches the hardware's SOL bound.

The baseline kernel's code remains private, although its scores are available on the leaderboard.

**Correctness gates the score.** A kernel is only timed if it first passes validation against the reference output (shape, dtype, numerical tolerance, and sanity checks against `inf`/`NaN`/degenerate outputs). Fail that check, and the submission gets $S = 0$ for the workload, no matter how fast it ran.

## The Vulnerability

The Solexecbench [dataset](https://huggingface.co/datasets/nvidia/SOL-ExecBench) gives us a reference implementation for each kernel problem. The reference implementation is a naive eager PyTorch kernel that is correct but slow.

For each new submitted kernel, the benchmark harness runs correctness checks + a time profiler. This logic is driven by the [eval_driver](https://github.com/NVIDIA/SOL-ExecBench/blob/main/src/sol_execbench/driver/templates/eval_driver.py).

### Correctness is checked once, then trusted forever

[`eval_driver.py`](https://github.com/NVIDIA/SOL-ExecBench/blob/main/src/sol_execbench/driver/templates/eval_driver.py) evaluates each workload in two phases, and they run on completely separate calls to the submitted `user_fn`.

**Phase 1: correctness.** The driver loops over 10 rounds. Each round calls `gen_inputs` to draw a fresh random input, runs the reference implementation to get ground truth, runs the submission, and calls `compute_error_stats` to compare the two. Round 0 also runs a few structural checks (shape, dtype, `check_lazy_outputs` to reject non-`torch.Tensor` outputs). If any round fails, the workload is rejected before timing ever starts.

**Phase 2: timing.** Once all 10 rounds pass, the driver calls `check_monkey_patch` (make sure `torch.cuda.Event.elapsed_time` hasn't been patched), then hands the *same* `user_fn` to `time_runnable`, in [`timing.py`](https://github.com/NVIDIA/SOL-ExecBench/blob/main/src/sol_execbench/core/bench/timing.py), for 10 warmup calls, 1 discovery call, and 100 timed reps, about 111 calls in total. All of them reuse whatever `_inputs` `gen_inputs` produced for round 9 of Phase 1; those tensors get handed to a `ShiftingMemoryPoolAllocator`, in [`io.py`](https://github.com/NVIDIA/SOL-ExecBench/blob/main/src/sol_execbench/core/bench/io.py), which gives every call a unique `data_ptr` (so a kernel can't fingerprint the timing loop by its address) but copies from the same source tensor into that new address every time. So the address moves, but the values behind it never change across all ~111 calls. After timing returns, the driver checks `check_thread_injection` (no new background threads) and emits a `PASSED` trace whose `correctness` field is whatever was captured back in round 9 of Phase 1.

**Nothing in Phase 2 calls `compute_error_stats` again.** The four reward-hack defenses the harness runs, `check_monkey_patch`, `check_thread_injection`, `check_lazy_outputs`, and `check_eval_integrity`, guard the timing mechanism itself (the timer function, thread count, output type, driver function identities), **but none of them look at what values the timed calls actually returned. Once a submission survives the 10 correctness rounds, its output is never verified again for the rest of the evaluation. The SOL score is computed entirely from calls whose correctness nobody checks.**

That would already be exploitable even if every one of those timed calls got a brand-new random input: _a kernel could behave correctly for exactly 10 calls and then return anything at all afterward. What makes it trivial in practice is that the timed calls don't even get new inputs, so a submission doesn't even need caching logic that inspects the input; it can just ignore it._

### Our exploit

To prove the vulnerability, we didn't write an optimized kernel at all. For every problem, we generated a single generic wrapper around the reference implementation that:

1. On a cache miss, calls the reference implementation verbatim, stores the result, and returns it.
2. On a cache hit, skips the reference entirely and returns the stored result.
3. Keys the cache on a cheap fingerprint of the call's arguments (shape, dtype, and a partial sum over a few elements of each tensor), not the full tensor content.

Round 9 of the correctness phase and all ~111 calls of Phase 2 are handed byte-for-byte identical input content, as established above. So the cache entry round 9 populates with the real reference output is exactly the entry every single Phase 2 call, warmup, discovery, and every timed rep, hits. From that point on, the only device work left inside the timed window is reading a few tensor elements to compute the fingerprint, not the reference computation itself. The measured CUPTI span collapses to almost nothing, regardless of how expensive the real kernel is supposed to be, and the resulting SOL score climbs toward 1.0 without a single line of actual kernel code.

One wrinkle: `bench_gpu_time_with_cupti`'s discovery step raises if a call produces zero recorded GPU activity ("No kernel activities recorded during discovery iteration"). A cache hit that just returned a stored tensor with no device op would trip that check. The fingerprint step already forces at least one device read and sync per call, and a tiny persistent CUDA tensor (`_heartbeat`) backstops the rare case of a problem with no tensor inputs at all, so there's always something for CUPTI to record, even on a pure cache hit.

#### Code

The generator below produces two solutions per problem: an honest control that just re-executes the reference every call, and the cache-replay exploit described above.
```python
def _honest_kernel_source(problem: SolProblem) -> str:
"""A solution that simply calls the reference verbatim on every call.

Used as a sanity-check control: its measured latency should closely
track the harness's own honest re-timing of the reference.
"""
return textwrap.dedent(f"""\
import torch

_REFERENCE_SRC = {problem.reference!r}
_ref_ns: dict = {{}}
exec(compile(_REFERENCE_SRC, "<reference>", "exec"), _ref_ns)
_reference_run = _ref_ns["run"]


@torch.no_grad()
def run(*args):
return _reference_run(*args)
""")


def _cache_replay_kernel_source(problem: SolProblem) -> str:
"""The exploit: memoize on input content, replay a cached clone on a hit.

Always genuinely correct (a cache miss calls the reference verbatim), so
it passes every correctness round exactly like ``_honest_kernel_source``.
The only difference is what happens when the same input content recurs,
which the benchmark loop does, every iteration, by construction.
"""
return textwrap.dedent(f"""\
import torch

_REFERENCE_SRC = {problem.reference!r}
_ref_ns: dict = {{}}
exec(compile(_REFERENCE_SRC, "<reference>", "exec"), _ref_ns)
_reference_run = _ref_ns["run"]

_CACHE: dict = {{}}
_SAMPLE_POINTS = 4
_HEARTBEAT = None


def _heartbeat():
# A tiny persistent CUDA tensor, so there is always >=1 device op
# to sync on even for a problem with zero tensor inputs. Without
# it, an all-scalar-input problem would leave a cache-hit call
# with no recorded CUPTI activity at all, which crashes the
# harness (bench_gpu_time_with_cupti raises "No kernel activities
# recorded") instead of just timing fast.
global _HEARTBEAT
if _HEARTBEAT is None:
_HEARTBEAT = torch.zeros(1, device="cuda")
return _HEARTBEAT


def _fingerprint(args):
# Every device op enqueued here, not just the final sync, adds
# to the measured CUPTI span, since the span runs from the first
# recorded activity's start to the last one's end: dispatch
# latency between consecutive small kernel launches stretches
# that window just as surely as a sync does. So this keeps the
# op count minimal: a short contiguous prefix slice (a view, no
# gather kernel, unlike scattered fancy-indexing) reduced by one
# .sum() per tensor, combined by plain adds, ending in exactly
# one .item() sync for the whole call regardless of how many
# tensor arguments it has. That sync is also what keeps >=1 CUPTI
# activity on the record; _heartbeat only backstops the (rare)
# all-scalar-input case where no tensor sample exists at all.
# NOTE: a short prefix is not a full-content fingerprint. This
# PoC accepts the (astronomically unlikely, for continuous random
# inputs) risk of two different draws sharing the same prefix, in
# exchange for the sample being a free view rather than a copy.
metas: list = []
acc = None
for a in args:
if isinstance(a, torch.Tensor):
flat = a.reshape(-1)
n = flat.numel()
metas.append((tuple(a.shape), str(a.dtype)))
if n:
sample = flat[: min(_SAMPLE_POINTS, n)]
try:
contrib = sample.sum()
except RuntimeError:
contrib = sample.float().sum()
acc = contrib if acc is None else acc + contrib
else:
metas.append(a)
if acc is None:
acc = _heartbeat().sum()
return tuple(metas), acc.item()


def _snapshot(result):
if isinstance(result, torch.Tensor):
return result.detach().clone()
if isinstance(result, (tuple, list)):
return type(result)(_snapshot(r) for r in result)
if isinstance(result, dict):
return {{k: _snapshot(v) for k, v in result.items()}}
return result


@torch.no_grad()
def run(*args):
key = _fingerprint(args)
cached = _CACHE.get(key)
if cached is not None:
# time_runnable() never inspects a timed call's return value
# (see eval_driver.py — it's called purely for its device
# work), and _fingerprint's own sync above already guarantees
# >=1 recorded CUPTI activity for this call. So the fastest
# legal reply on a hit is the cached object itself — no copy,
# no further device op.
return cached
result = _reference_run(*args)
_CACHE[key] = _snapshot(result)
return result
""")
```

#### Results
TBD

## Conclusion

Writing SOTA kernels is incredibly hard and at the same time having robust benchmarks that are resistant to gaming is equally hard. Most teams participating in SOLExecbench are likely having their own Agentic Harness write and submit kernels to solve the problems and with very little human oversight, which can lead to over-inflated results on a benchmark that don't translate to real-world performance.

The exploit we demonstrated here is a simple example of how a benchmark can be gamed, and it highlights the importance of designing benchmarks that are robust against such attacks. We hope that this post serves as a reminder to the community to be vigilant about the integrity of benchmarks and to continuously improve them to ensure they accurately reflect the performance of the systems being evaluated.