Skip to content

bench: run the full compress suite under --gpu-decompress - #9399

Draft
joseph-isaacs wants to merge 9 commits into
developfrom
claude/gpu-compress-benchmarks-x788xc
Draft

bench: run the full compress suite under --gpu-decompress#9399
joseph-isaacs wants to merge 9 commits into
developfrom
claude/gpu-compress-benchmarks-x788xc

Conversation

@joseph-isaacs

Copy link
Copy Markdown
Contributor

Rationale for this change

--gpu-decompress in compress-bench was gated behind an incremental allow-list that
contained a single entry (TPC-H l_comment canonical), so the GPU compression benchmark
job only ever measured one dataset. This widens GPU coverage to the whole compress suite,
including the airquality dataset.

Opened as a draft to let the GPU benchmark job tell us which datasets actually have
working CUDA decode kernels — that verification is exactly what the allow-list was
standing in for, and it cannot be done locally without a GPU.

What changes are included in this PR?

  • Remove the gpu_decompress_benchmarks allow-list from benchmarks/compress-bench/src/main.rs.
    With no --datasets filter, --gpu-decompress now runs every dataset in the suite:
    taxi, the Public BI sets (Arade, Bimbo, CMSprovider, Euro2016, Food, HashTags), both
    TPC-H l_comment variants, the StructListOfInts variants, plus airquality and rplace.
  • The CPU default path is unchanged and still skips airquality/rplace. Note that the GPU
    path now downloads both from pcodec's public S3 bucket on every run; the existing CPU-side
    exclusion exists to avoid creating egress charges for pcodec, so this may want caching
    rather than a permanent CI cost.
  • Update benchmarks/compress-bench/README.md to match.

What APIs are changed? Are there any user-facing changes?

No library API changes. The only behavior change is to the compress-bench benchmark binary:
--gpu-decompress no longer filters the dataset list.

Checks

  • cargo clippy -p compress-bench --all-targets passes.
  • cargo +nightly fmt could not be run in this environment (the nightly toolchain ICEs);
    the edits are formatting-neutral but CI fmt is unverified.
  • The GPU run itself is unverified locally — no CUDA device available. That is what this
    draft's benchmark job is for.

Generated by Claude Code

claude added 2 commits August 13, 2026 12:17
Drop the incremental GPU allow-list so --gpu-decompress runs every dataset
in the compress suite, and include the airquality dataset on the GPU path.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
airquality and rplace come from the same pcodec bucket, so gate them
consistently: the GPU suite runs every compress dataset.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@joseph-isaacs joseph-isaacs added the action/bench-gpu-compress Run only the GPU compression benchmark on this PR label Aug 13, 2026 — with Claude
@github-actions github-actions Bot removed the action/bench-gpu-compress Run only the GPU compression benchmark on this PR label Aug 13, 2026
@codspeed-hq

codspeed-hq Bot commented Aug 13, 2026

Copy link
Copy Markdown

Merging this PR will degrade performance by 12.14%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚡ 1 improved benchmark
❌ 3 regressed benchmarks
✅ 2033 untouched benchmarks
⏩ 46 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
WallTime words_gather_dispatch[1024] 9 ns 13 ns -30.77%
WallTime words_gather_scalar[65536] 8.3 µs 9.4 µs -11.83%
Simulation take[small_m/shuffled/primitive/nonnull/chunks=16384/indices=16] 1 ms 1.2 ms -11.26%
WallTime words_gather_dispatch[1024] 33 ns 30 ns +10%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing claude/gpu-compress-benchmarks-x788xc (cf631cd) with develop (1a493b6)

Open in CodSpeed

Footnotes

  1. 46 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@github-actions

github-actions Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

BENCHMARK FAILED

GPU Compression failed. Check the workflow run for details.

The CUDA DateTimeParts executor dispatched with match_each_signed_integer_ptype,
so it panicked with "Unsupported ptype u16" on the taxi benchmark. Compression
picks the narrowest ptype per component, and the CPU decoder already accepts any
integer ptype, so match the CPU behaviour and generate the kernel for every
signed and unsigned integer width.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@joseph-isaacs joseph-isaacs added the action/bench-gpu-compress Run only the GPU compression benchmark on this PR label Aug 13, 2026 — with Claude
@github-actions github-actions Bot removed the action/bench-gpu-compress Run only the GPU compression benchmark on this PR label Aug 13, 2026
@joseph-isaacs joseph-isaacs added the changelog/fix A bug fix label Aug 13, 2026 — with Claude
claude added 2 commits August 13, 2026 17:01
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
RunEnd GPU decoding bailed when values carried Validity::Array. The error
message claimed a CPU fallback, but execute_cuda can only fall back while all
buffers are host-resident, so a device-resident scan (as in the GPU compression
benchmark) failed outright.

Add a runend_bool kernel that expands the per-run validity bitmap through the
same run mapping as the values. Each thread owns a whole output byte so threads
never race on bits within a byte.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@joseph-isaacs joseph-isaacs added the action/bench-gpu-compress Run only the GPU compression benchmark on this PR label Aug 13, 2026 — with Claude
@github-actions github-actions Bot removed the action/bench-gpu-compress Run only the GPU compression benchmark on this PR label Aug 13, 2026
The Public BI datasets are converted from CSV with the DuckDB CLI, which the
CPU benchmark runner already installs. The GPU job never needed it while its
suite was a single TPC-H dataset; now that it runs the full suite, Arade fails
with "No such file or directory" when spawning duckdb.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@joseph-isaacs joseph-isaacs added the action/bench-gpu-compress Run only the GPU compression benchmark on this PR label Aug 13, 2026 — with Claude
@github-actions github-actions Bot removed the action/bench-gpu-compress Run only the GPU compression benchmark on this PR label Aug 13, 2026
The GPU compression benchmark failed on Euro2016 with "No CUDA kernel for
encoding vortex.masked". A MaskedArray is a child array that carries no nulls
of its own plus the validity bitmap that supplies them, so decode the child on
the GPU, decode the mask on the GPU, and attach the mask to the result.

Bail when the child itself carries a per-element validity bitmap: intersecting
two device-resident bitmaps would need a CPU compute pass, and MaskedArray's
own invariant makes that case unreachable for well-formed arrays.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@joseph-isaacs joseph-isaacs added the action/bench-gpu-compress Run only the GPU compression benchmark on this PR label Aug 13, 2026 — with Claude
@github-actions github-actions Bot removed the action/bench-gpu-compress Run only the GPU compression benchmark on this PR label Aug 13, 2026
claude added 2 commits August 13, 2026 17:47
Canonical arrays keep validity as a separate child array, and into_host moved
only the data buffer, leaving a device-resident bitmap that CPU compute cannot
read ("unwrap_host called for Device allocation"). Any executor returning
Validity::Array from the GPU hit this; the MaskedExecutor test caught it.

Also call the executors directly in the nullable RunEnd tests. Going through
execute_cuda silently falls back to CPU for a host-resident array, so those
tests passed without ever running the new runend_bool kernel.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…tible

vortex.sparse has no CUDA decode kernel, and only_cuda_compatible already
excluded the integer and float sparse schemes. The string variant was missed,
so Euro2016 failed the GPU compression benchmark with "No CUDA kernel for
encoding vortex.sparse".

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@joseph-isaacs joseph-isaacs added the action/bench-gpu-compress Run only the GPU compression benchmark on this PR label Aug 13, 2026 — with Claude
@github-actions github-actions Bot removed the action/bench-gpu-compress Run only the GPU compression benchmark on this PR label Aug 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/fix A bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants