Skip to content

fix: preserve no-filter SMJ matches across pending outer batches#23049

Open
neilconway wants to merge 2 commits into
apache:mainfrom
neilconway:neilc/fix-smj-pending-cross-batches
Open

fix: preserve no-filter SMJ matches across pending outer batches#23049
neilconway wants to merge 2 commits into
apache:mainfrom
neilconway:neilc/fix-smj-pending-cross-batches

Conversation

@neilconway

@neilconway neilconway commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

When the no-filter bitwise sort-merge join path finds a matching key,
it advances the inner cursor past that key before marking all matching
outer rows. If the outer key group continues into the next outer batch
and polling that batch returns Pending, poll_join resumes from its
top-level state with the inner cursor already past the matched key.

On resume, the stream still retained the already-emitted outer batch.
That stale batch caused the pending boundary state to be applied to the
wrong batch and then discarded. When the actual next outer batch was
loaded later, rows continuing the matched key could compare as Less than
the current inner key and be incorrectly treated as unmatched.

Fix this by clearing outer_batch after emitting the fully consumed batch
and before polling for the next outer batch. This makes the resumed
top-level state load the actual next outer batch before applying
resume_boundary, matching the filtered code path.

What changes are included in this PR?

  • Bug fix for SMJ semi/anti-join behavior across outer batches
  • Add unit test

Are these changes tested?

Yes.

Are there any user-facing changes?

No.

@github-actions github-actions Bot added the physical-plan Changes to the physical-plan crate label Jun 19, 2026
@neilconway neilconway force-pushed the neilc/fix-smj-pending-cross-batches branch from 895944d to 421e249 Compare June 19, 2026 15:54
@neilconway

Copy link
Copy Markdown
Contributor Author

cc @mbutrovich @pantShrey

@pantShrey

Copy link
Copy Markdown
Contributor

Thank you for catching this and for the clear explanation. I believe the fix is correct. setting outer_batch = None before the poll makes the no-filter path consistent with the filtered path, which already had this pattern. The test covers the exact failure scenario. LGTM.

@viirya viirya left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice fix — I traced the state machine and the bug is exactly as described.

The single added line restores symmetry with the filtered path, which already does the same thing at the matching point in process_filtered_match_loop (with the comment "Clear stale batch before polling"). Without self.outer_batch = None; here, after poll_next_outer_batch returns Pending and poll_join re-enters from the top, the if self.outer_batch.is_none() guard is skipped because the stale already-emitted batch is still set. resume_boundary then compares saved_keys against the old batch's keys, applies the boundary state to the wrong batch, and the continuation rows of the matched key group in the real next batch get treated as unmatched. Clearing it first forces the top-level loop to load the actual next batch before resume_boundary runs.

I verified this locally:

  • The new no_filter_boundary_pending_with_unmatched_prefix test passes with the fix.
  • Reverting just the one-line fix makes it fail with LeftSemi left: [1], right: [1, 2] — the continuation row is dropped — so the test is a genuine regression guard, not a tautology.
  • The full sort_merge_join suite still passes (62 tests).

One optional nit: consider adding the same one-line comment the filtered path has (// Clear stale batch before polling) so the parallel between the two paths is obvious to future readers.

LGTM.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

physical-plan Changes to the physical-plan crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI failure: fuzz_cases::join_fuzz::test_left_anti_join_1k

3 participants