Skip to content

feat(vector): complete data evolution support - #7

Open
ChaomingZhangCN wants to merge 2 commits into
mainfrom
cursor/vector-data-evolution-8e06
Open

feat(vector): complete data evolution support#7
ChaomingZhangCN wants to merge 2 commits into
mainfrom
cursor/vector-data-evolution-8e06

Conversation

@ChaomingZhangCN

@ChaomingZhangCN ChaomingZhangCN commented Aug 24, 2026

Copy link
Copy Markdown
Owner

Purpose

Linked issue: apache#197

Implement Phase 2 VECTOR data-evolution support for append-only Parquet tables:

  • allow VECTOR fields in data-evolution tables;
  • add a validated SchemaManager commit path for add/drop evolution;
  • preserve VECTOR reads across historical schema files and partial-column updates;
  • reject incompatible VECTOR dimension or element-type changes before persisting the next schema;
  • reject VECTOR comparator roles including sequence and sequence-group ordering fields, while retaining primary, partition, and bucket-key validation.

Tests

  • Built paimon-common-test, paimon-core-test, paimon-data-evolution-table-test, paimon-write-and-read-inte-test, and paimon-blob-table-inte-test
  • Related schema/VECTOR unit suites: 96 passed
  • Complete Data Evolution integration suite: 198 passed, 6 expected skips
  • Complete write/read integration suite: 144 passed
  • Complete BLOB/Data Evolution integration suite: 162 passed
  • python3 -m pre_commit run --files <changed-files>
  • git diff --check origin/main...HEAD

API and Format

No public API, storage-format, or protocol changes. VECTOR continues using the existing Parquet LIST representation. The new schema-commit entry point is internal to the core schema manager.

Documentation

Updated the VECTOR data-type documentation with validated data-evolution behavior and comparator restrictions.

Generative AI tooling

Generated-by: Cursor Agent GPT-5.6 Sol

Open in Web Open in Cursor 

cursoragent and others added 2 commits August 24, 2026 02:38
Co-authored-by: 小明同学 <ChaomingZhangCN@users.noreply.github.com>
Co-authored-by: 小明同学 <ChaomingZhangCN@users.noreply.github.com>
@ChaomingZhangCN
ChaomingZhangCN marked this pull request as ready for review August 24, 2026 06:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants