Skip to content

[FLINK-40223][runtime] Support additional Flink SQL string functions in YAML Transform - #4488

Open
haruki-830 wants to merge 3 commits into
apache:masterfrom
haruki-830:FLINK-40223
Open

[FLINK-40223][runtime] Support additional Flink SQL string functions in YAML Transform#4488
haruki-830 wants to merge 3 commits into
apache:masterfrom
haruki-830:FLINK-40223

Conversation

@haruki-830

Copy link
Copy Markdown
Contributor

Summary

This commit expands the string function coverage of Flink CDC YAML Transform by adding commonly used Flink SQL string manipulation, search, predicate, and encoding functions. These functions can now be used in row-level transform projections and filters.

Key Changes

  1. Additional String Function Support
  • Added LTRIM, RTRIM, BTRIM, OVERLAY, POSITION, LOCATE, and INSTR.
  • Added CONCAT_WS, LPAD, RPAD, REPLACE, REPEAT, LEFT, and RIGHT.
  • Added STARTSWITH, ENDSWITH, TO_BASE64, and FROM_BASE64.
  • Handles null inputs, Unicode code points, and function-specific boundary cases consistently with Flink SQL semantics.
  1. Parser and Runtime Integration
  • Registered the new functions with their operand and return type definitions in the transform operator table.
  • Added parser handling for SQL-specific OVERLAY and POSITION syntax.
  • Integrated the functions with Janino expression compilation for runtime evaluation.
  1. Test Coverage and Documentation
  • Added unit tests for function behavior, null handling, Unicode strings, and boundary cases.
  • Added parser and transform runtime tests for valid and invalid expressions.
  • Added YAML specification coverage for projection results, including matched, unmatched, and null inputs.
  • Documented the new functions in both the English and Chinese transform documentation.

JIRA Reference

https://issues.apache.org/jira/browse/FLINK-40223

@github-actions github-actions Bot added docs Improvements or additions to documentation composer runtime labels Jul 24, 2026
@haruki-830
haruki-830 marked this pull request as ready for review July 24, 2026 10:02
@haruki-830

Copy link
Copy Markdown
Contributor Author

@lvyanquan @yuxiqian PTAL, thanks!

@yuxiqian yuxiqian left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Haruki for the nice work, just left some minor comments.

Comment thread docs/content.zh/docs/core-concept/transform.md Outdated
@haruki-830

Copy link
Copy Markdown
Contributor Author

@yuxiqian Thanks for the review! I addressed the comments in the lastest commit.

@yuxiqian yuxiqian left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 from my side (except one minor comment on test specs). Need another eye from Kunni.

Comment thread flink-cdc-composer/src/test/resources/specs/string.yaml
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

composer docs Improvements or additions to documentation runtime

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants