[fix](fe) Fix stream scan normalization and snapshot/reset pruning#65468
Open
yujun777 wants to merge 7 commits into
Open
[fix](fe) Fix stream scan normalization and snapshot/reset pruning#65468yujun777 wants to merge 7 commits into
yujun777 wants to merge 7 commits into
Conversation
…and normalize after CTE inline Two fixes for LogicalOlapTableStreamScan handling in Nereids planner: 1. Override withVirtualColumns to return LogicalOlapTableStreamScan instead of LogicalOlapScan, preventing type downgrade during LogicalPlanDeepCopier.deepCopy used by CTEInline. 2. Move NormalizeOlapTableStreamScan execution to after CTE inline so StreamScans exposed by CTE expansion get normalized. Previously it ran inside notTraverseChildrenOf(CTEAnchor) which blocked traversal into CTE producers. This fixes CTE+stream hidden column queries and IVM incremental refresh failures caused by un-normalized stream scans.
Issue Number: close apache#65339 Related PR: Problem Summary: Normalize stream scan rules were executed in a late CTE-related rewrite topic, which could be skipped in trees with CTE-related traversal constraints, causing delta/rewrite paths to miss synthetic change_type/version project aliases and fail when buildDmlFactorExpr accessed hidden stream columns. This moves into the early table/physical optimization topic after partition pruning and removes the separate late topic from , and updates regression baseline for to the fixed incremental result behavior. None - Test: Regression test / Unit Test / Manual test / No need to test (with reason) - test_ivm_basic_mtmv (baseline .out updated) - Behavior changed: Yes (CTE stream-scan normalization now runs in the proper rewrite phase) - Does this need documentation: No
yujun777
requested review from
924060929,
englefly,
morrySnow and
starocean999
as code owners
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
yujun777
changed the title
This was referenced Jul 10, 2026
Contributor
Author
|
run buildall |
yujun777
changed the title
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29280 ms |
Contributor
TPC-DS: Total hot run time: 180182 ms |
Contributor
ClickBench: Total hot run time: 25.04 s |
yujun777
added a commit
to yujun777/doris
that referenced
this pull request
Jul 10, 2026
### What problem does this PR solve? Issue Number: close apache#65469 Related PR: apache#65468 Problem Summary: Several IVM aggregate regression cases used `REFRESH ... COMPLETE` before validating later incremental behavior. COMPLETE refresh does not advance the stream consumption offset, so the following `REFRESH ... INCREMENTAL` could re-consume historical changes and produce results that no longer matched the intended test point. This change updates the affected aggregate suites to establish the baseline with an initial incremental refresh, keeps the later COMPLETE refreshes that compare against snapshot ground truth, and regenerates the `.out` files so the expected results match the corrected stream-offset behavior. ### Release note None ### Check List (For Author) - Test: Regression test - `./run-regression-test.sh --run -d mtmv_p0 -s test_ivm_agg_2,test_ivm_agg_3,test_ivm_agg_4,test_ivm_agg_6 -forceGenOut --no-parallel` - Behavior changed: Yes (the affected regression cases now validate incremental results after establishing stream offsets with incremental refresh) - Does this need documentation: No
yujun777
added a commit
to yujun777/doris
that referenced
this pull request
Jul 10, 2026
### What problem does this PR solve? Issue Number: close apache#65469 Related PR: apache#65468 Problem Summary: Several IVM aggregate regression cases used `REFRESH ... COMPLETE` before validating later incremental behavior. COMPLETE refresh does not advance the stream consumption offset, so the following `REFRESH ... INCREMENTAL` could re-consume historical changes and produce results that no longer matched the intended test point. This change updates the affected aggregate suites to establish the baseline with an initial incremental refresh, keeps the later COMPLETE refreshes that compare against snapshot ground truth, and regenerates the `.out` files so the expected results match the corrected stream-offset behavior. ### Release note None ### Check List (For Author) - Test: Regression test - `./run-regression-test.sh --run -d mtmv_p0 -s test_ivm_agg_2,test_ivm_agg_3,test_ivm_agg_4,test_ivm_agg_6 -forceGenOut --no-parallel` - Behavior changed: Yes (the affected regression cases now validate incremental results after establishing stream offsets with incremental refresh) - Does this need documentation: No
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: Add a regression case for min_delta stream queries that project the stream change type column through a CTE and reuse that CTE twice in a self-join. The case verifies the output through an auto-generated .out file and keeps the existing stream regression coverage intact.
### Release note
None
### Check List (For Author)
- Test: Regression test
- ./run-regression-test.sh --run -d table_stream_p0 -s test_min_delta_stream
- Behavior changed: No
- Does this need documentation: No
Contributor
Author
|
run buildall |
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: Stream scan planning had split state for snapshot and reset, while partition pruning still depended on table-specific non-empty checks that did not receive stream read semantics. This made the stream planning path harder to reason about and caused snapshot/reset handling to diverge from incremental reads. This change replaces the dual boolean state on logical stream scan with a single StreamReadMode enum, threads the optional read mode through LogicalCatalogRelation into selectNonEmptyPartitionIds, and lets the stream wrapper choose incremental pruning from stream.hasData while delegating snapshot/reset pruning to the base table path. The result keeps the read-mode semantics continuous from binding through rewrite and pruning, while preserving the convenience isSnapshot/isReset/isIncremental helpers on the scan node.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- ./run-fe-ut.sh --run org.apache.doris.nereids.trees.plans.ExplainTableStreamPlanTest
- Behavior changed: Yes (stream scan snapshot/reset partition pruning now follows explicit read-mode plumbing instead of split boolean flags)
- Does this need documentation: No
yujun777
changed the title
Contributor
Author
|
run buildall |
### What problem does this PR solve? Issue Number: None Related PR: apache#65468 Problem Summary: This follow-up keeps the stream scan helper implementation order aligned with the surrounding builder methods by moving after in . The change is code organization only and does not alter semantics. ### Release note None ### Check List (For Author) - Test: No need to test (with reason) - Method order only, no logic change - Behavior changed: No - Does this need documentation: No
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29961 ms |
Contributor
TPC-DS: Total hot run time: 180801 ms |
Contributor
ClickBench: Total hot run time: 24.94 s |
| ImmutableSet.of(LogicalCTEAnchor.class), | ||
| () -> { | ||
| List<RewriteJob> rewriteJobs = Lists.newArrayListWithExpectedSize(300); | ||
| if (Config.enable_table_stream) { |
Contributor
There was a problem hiding this comment.
maybe move config check inside rule?
Contributor
Author
There was a problem hiding this comment.
The rule already matches only LogicalOlapTableStreamScan. If such a node exists, it should always be normalized regardless of Config.enable_table_stream. The config controls whether stream tables can be created, but it is not a condition for this normalization rule, so the config check is not needed here.
Contributor
Author
There was a problem hiding this comment.
Also, a rewrite rule first performs pattern matching and only executes its rewrite function after the pattern matches. If no LogicalOlapTableStreamScan exists, this rule is not matched by any plan node and its rewrite logic is never executed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #65469
Related PR:
Problem Summary: This PR fixes two FE issues in the stream scan path.
First, when a stream scan appears inside a CTE,
NormalizeOlapTableStreamScanmay not be applied on the reused CTE branch, so the stream virtual columns are not lowered consistently and later rewrite steps see the wrong scan shape.Second,
PruneEmptyPartitionusesselectNonEmptyPartitionIdsbefore stream scan normalization. Forstream@snapshotandstream@reset, the old stream-wrapper pruning logic still used incremental stream visibility semantics, so a stream with no new delta could be pruned as empty even though snapshot/reset should still read snapshot data.This change keeps stream read mode as explicit logical scan state, threads that mode through
LogicalCatalogRelationinto partition pruning, preserves stream scan helper predicates on the logical node, and lets snapshot/reset pruning follow the base table path while incremental reads still use stream delta visibility. It also keeps stream scan normalization behavior consistent when the scan is referenced through a CTE.Release note
None
Check List (For Author)
./run-fe-ut.sh --run org.apache.doris.nereids.trees.plans.ExplainTableStreamPlanTeststream@snapshot/stream@resetare no longer pruned by incremental stream visibility)