Skip to content

[python] Support row-count based file rolling for data-evolution append tables#8863

Open
XiaoHongbo-Hope wants to merge 5 commits into
apache:masterfrom
XiaoHongbo-Hope:de_row_num_rolling
Open

[python] Support row-count based file rolling for data-evolution append tables#8863
XiaoHongbo-Hope wants to merge 5 commits into
apache:masterfrom
XiaoHongbo-Hope:de_row_num_rolling

Conversation

@XiaoHongbo-Hope

Copy link
Copy Markdown
Contributor

Purpose

#8739 added row-count based data file rolling (target-file-row-num) on the core side. PyPaimon file-store writers previously did not support it and failed fast whenever the option was enabled.

This PR implements target-file-row-num rolling for data-evolution append tables in PyPaimon.

Changes

  • DataWriter._check_and_roll_if_needed: roll a file when it reaches target-file-row-num rows or target-file-size, whichever comes first (exact slicing, mirroring the format-table path). Defaults to the max long (disabled), so size-based rolling is unchanged when the option is unset.
  • FileStoreWrite._create_data_writer: narrow the fail-fast so the option is permitted only for data-evolution append tables (i.e. AppendOnlyDataWriter — not primary-key, blob or vector writers, which override rolling and remain unsupported for now).

Tests

data_evolution_row_rolling_test.py:

  • rolls when the row count exceeds the limit (10 rows / limit 3 → files of 3,3,3,1);
  • exact multiple rolls evenly;
  • below the limit stays a single file;
  • unset option does not roll by rows;
  • a non-DE table still fails fast.

@XiaoHongbo-Hope
XiaoHongbo-Hope marked this pull request as ready for review July 27, 2026 13:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant