[core] Introduce vector-store for data-evolution table

[ColdL]

Purpose

Linked issue: update #7011

The goal of this PR is to optimize storage layout for vector scenarios in the Data Evolution table, specifically by storing vector columns and potentially associated columns in specified file formats.

For example, scalar columns can be stored using Parquet format, while vector columns and columns that may require point lookups during vector search can be stored using file formats like Lance.

1. Configuration

This PR introduces three new configuration options:

• vector-field: defines the column names for separate storage
• vector.file.format: defines the file format
• vector.target-file-size: specifies the file size threshold for rolling

2. Storage Layout

When this feature is enabled, a set of columns specified by vector-field will be stored separately in the file format specified by vector.file.format, marked by .vector-store. in the data file path.

File Path Pattern: data-xxx-{count}.vector-store.{file-format}

This design serves two purposes:

• (1) The .vector-store. segment identifies these as separately stored column groups
• (2) The trailing .{file-format} follows current conventions, using the file format as the suffix

Note: Perhaps .vector. is better than .vector-store., if confirmed, I will update this accordingly. Please see the discussion below for details.

The final storage layout might be:

• data-xxx-0.parquet
• data-xxx-1.blob
• data-xxx-2.vector-store.lance
• data-xxx-3.vector-store.lance
• data-xxx-4.vector-store.lance

These vector-store files are associated with regular columns through Row-tracking / Data Evolution.

Tests

API and Format

Documentation

[leaves12138]

How about these names? @JingsongLi

[ColdL]

Fixed. Now the config names in code are consistent with the public configuration keys.

[leaves12138]

LGTM
Thanks for @ColdL , can you rebase the latest master to resolve the conflict

[JingsongLi]

I think the PR statement should clearly state a few things:

1. What configuration determines the separate storage of Vector? We need to design this configuration.
2. How to name the Vector file, perhaps it would be better to end with .vector.

[JingsongLi]

You can also create a separate doc in paimon/docs.

[ColdL]

I think the PR statement should clearly state a few things:

1. What configuration determines the separate storage of Vector? We need to design this configuration.
2. How to name the Vector file, perhaps it would be better to end with .vector.

@JingsongLi Thanks for the review! I've updated the PR description. After confirmation, I will continue to update and add the corresponding docs.

[JingsongLi]

@ColdL How about:

1. vector.file.format => by default none, if configured, store vector separately in file.
2. file name just use xxx.vector.lance, consistent with blob files.

[ColdL]

@ColdL How about:

1. vector.file.format => by default none, if configured, store vector separately in file.
2. file name just use xxx.vector.lance, consistent with blob files.

Got it. Do you think we should retain the vector-field option? The initial thought was that, in retrieval scenarios, output columns could still benefit from file-format-level optimizations.

[JingsongLi]

vector-field

vector-field is for Flink & Spark engines without Vector Type. So I think it should be retained.