Skip to content

fix(fsst): adaptive i32→i64 codes_offsets promotion#7851

Draft
mprammer wants to merge 1 commit intodevelopfrom
mp/fsst-i32-overflow-adaptive
Draft

fix(fsst): adaptive i32→i64 codes_offsets promotion#7851
mprammer wants to merge 1 commit intodevelopfrom
mp/fsst-i32-overflow-adaptive

Conversation

@mprammer
Copy link
Copy Markdown

@mprammer mprammer commented May 8, 2026

Fixes #7833. fsst_compress_iter previously hardcoded VarBinBuilder::<i32> for the FSST output and panicked once cumulative compressed bytes crossed i32::MAX. The companion change to vortex-array/src/arrays/varbin/compute/compare.rs handles the latent Arrow LargeBinary == Binary mismatch that becomes reachable once any FSST output uses i64 offsets.

The codes-offsets buffer now starts as i32 and promotes to i64 on the first append where data.len() crosses i32::MAX — one O(n) copy of existing offsets, sub-millisecond at any realistic n. Common case is identical to the original i32 path. Tests: a small-input round-trip asserts the i32 baseline; a CI-gated ~4.5 GiB test exercises the i64 path end-to-end.

🤖 Generated with Claude Code

Single-pass; common case keeps the compact i32 layout, pathological inputs transparently get i64.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: mprammer <martin@spiraldb.com>
@codspeed-hq
Copy link
Copy Markdown

codspeed-hq Bot commented May 8, 2026

Merging this PR will degrade performance by 24.95%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 2 improved benchmarks
❌ 6 regressed benchmarks
✅ 1200 untouched benchmarks

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation take_map[(0.1, 0.5)] 1,111.5 µs 988.1 µs +12.49%
Simulation take_map[(0.1, 1.0)] 1.8 ms 1.6 ms +11.89%
Simulation new_bp_prim_test_between[i32, 16384] 94.9 µs 110 µs -13.73%
Simulation new_bp_prim_test_between[i32, 32768] 141.2 µs 170.8 µs -17.31%
Simulation new_bp_prim_test_between[i64, 16384] 115.5 µs 145.4 µs -20.61%
Simulation new_bp_prim_test_between[i64, 32768] 178.3 µs 237.6 µs -24.95%
Simulation new_bp_prim_test_between[i16, 32768] 120.6 µs 135.1 µs -10.71%
Simulation new_alp_prim_test_between[f64, 16384] 127.3 µs 149.8 µs -15.04%

Comparing mp/fsst-i32-overflow-adaptive (9e2bcdc) with develop (ff12040)

Open in CodSpeed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

FSST: fsst_compress panics on cumulative output >2 GiB (i32 offset overflow)

1 participant