Skip to content

Flink: Fix read split table properties being ignored - #17445

Open
vishnuprakaz wants to merge 2 commits into
apache:mainfrom
vishnuprakaz:flink-read-split-table-props
Open

Flink: Fix read split table properties being ignored#17445
vishnuprakaz wants to merge 2 commits into
apache:mainfrom
vishnuprakaz:flink-read-split-table-props

Conversation

@vishnuprakaz

Copy link
Copy Markdown
Contributor

Setting read.split.target-size, read.split.planning-lookback, or read.split.open-file-cost as a table property has no effect on Flink reads. The split options in FlinkReadOptions declare default values, so FlinkConfParser always returns the Flink default and never reaches the table property fallback in FlinkReadConf, even though the docs promise read option, then Flink configuration, then table property. This is a regression from #5967, which introduced the read options with declared defaults; before that the planner only set the split scan options when explicitly configured and the table properties were honored.

The split options now declare no default, like the write options, and the defaults are applied in FlinkReadConf after the table property lookup. Needs a backport to 1.20 and 2.0

@github-actions github-actions Bot added the flink label Jul 31, 2026
conf = new FlinkReadConf(table, ImmutableMap.of(), new Configuration());
assertThat(conf.splitFileOpenCost()).isEqualTo(TableProperties.SPLIT_OPEN_FILE_COST_DEFAULT);
}
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new test exercises only three of the four precedence tiers (flinkConf > tableProperty > default) for each split option; the top tier; an explicit entry in the readOptions map (the second FlinkReadConf constructor argument) is never asserted. All three tests pass ImmutableMap.of() as the readOptions arg, so the test suite would pass even if the read-option tier were completely broken. Add a fourth assertion per method: instantiate with e.g. ImmutableMap.of(FlinkReadOptions.SPLIT_SIZE, "999") and assert the result beats both the flinkConf value and the table property.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants