Skip to content

[AURON #2405] Add primitive fast path for single-column integer sort#2406

Open
yew1eb wants to merge 1 commit into
apache:masterfrom
yew1eb:issue_2405
Open

[AURON #2405] Add primitive fast path for single-column integer sort#2406
yew1eb wants to merge 1 commit into
apache:masterfrom
yew1eb:issue_2405

Conversation

@yew1eb

@yew1eb yew1eb commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #2405

Rationale for this change

For TPC-H Q21 (SF=10, lineitem = 59,986,052 rows), the right-side lineitem SortMergeJoin sorts on the single-column integer key l_orderkey (Int64). Auron's native SortExec was ~8.6x slower than Spark (13.8s vs 1.6s) because it always routes through RowConverter byte encoding, whereas Spark uses radix sort on an 8-byte prefix (O(n)). See #2405 for the full analysis.

What changes are included in this PR?

Type-aware fast path in ExternalSorter (native-engine/datafusion-ext-plans/src/sort_exec.rs):

  • PrimitiveSortKey::try_new() detects a single direct Column of a fixed-width integer type (Int8/16/32/64, UInt*, Date32/64, Timestamp).
  • When it applies, sort each batch via Arrow sort_to_indices on the raw array (honoring descending/nulls_first) and skip the RowConverter encode + byte-compare path. Downstream sorted_indices becomes an identity range and avoids an extra take_batch copy.
  • Two #[ignore] benchmarks in fuzztest comparing Auron vs DataFusion native.

Are there any user-facing changes?

No.

How was this patch tested?

Existing fuzztest_in_mem_sorting / fuzztest_external_sorting pass. Manual #[ignore] benchmark (1M Int64 rows, varying repetition):

Before (RowConverter path):
[sort in-mem] repeat= 1, auron=3.014s, datafusion=0.487s (6.18x)
[sort in-mem] repeat=100, auron=2.384s, datafusion=0.353s (6.75x)
[sort external] repeat=100, auron=2.313s, datafusion=0.339s (6.83x)

After (primitive fast path):
[sort in-mem] repeat= 1, auron=0.051s, datafusion=0.023s (2.25x)
[sort in-mem] repeat=100, auron=0.044s, datafusion=0.020s (2.20x)
[sort external] repeat=100, auron=0.043s, datafusion=0.030s (1.42x)

  • Before: Auron is ~6x slower than DataFusion (RowConverter encoding + byte-comparison overhead).
  • After: the gap narrows to ~2x, effective for both in-memory and external sorts.
  • The remaining gap comes from the K-way merge stage, which still uses RowConverter byte comparison — it can be further reduced with a PrimitiveKeyCollector.
  • DataFusion's native SortExec calls Arrow sort_to_indices → Rust sort_unstable_by to compare raw i6 values for primitives.
  • Spark's native SortExec additionally uses radix sort for single-column integers, which is theoretically faster than DataFusion's sort_unstable_by, so the Auron-vs-Spark gap is likely larger than 2x — implementing radix sort can close it further.

Was this patch authored or co-authored using generative AI tooling?

No.

@yew1eb
yew1eb force-pushed the issue_2405 branch 2 times, most recently from 293b02c to 3f07f0b Compare July 20, 2026 15:00
@yew1eb
yew1eb requested a review from richox July 21, 2026 02:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Performance] Native SortExec ~8x slower than Spark for single-column integer keys (TPC-H Q21 lineitem)

1 participant