[FLINK-36520] MySQL CDC Supports connecting to MySQL server 8.4+

[yuxiqian]

This closes FLINK-36520, FLINK-36590, FLINK-37503.

Currently, trying to use MySQL CDC with MySQL 8.4 would cause the following exception:

Caused by: org.apache.flink.util.FlinkRuntimeException: Read split MySqlSnapshotSplit{tableId=customer.customers, splitId='customer.customers:0', splitKeyType=[`id` BIGINT NOT NULL], splitStart=null, splitEnd=null, highWatermark=null} error due to org.apache.flink.util.FlinkRuntimeException: Cannot read the binlog filename and position via 'SHOW MASTER STATUS'. Make sure your server is correctly configured.

The reason is MySQL has changed some questionable words like MASTER and SLAVE to more appropriate ones, so some commands are no longer compatible. Legacy MySqlSource (based on Debezium) wasn't fixed since it's not possible to upgrade Debezium dependency for now, and it has been deprecated for a long time anyway.

Marked this PR as a draft, needs more discussion and testing over this.

[leonardBang]

Could you investigate binlog compress feature would been supported via this PR or not? It requires MySQL 8.0.20 above version IIRC.

[morozov]

Correct me if I'm wrong, but it looks like using the proper SQL command for showing binary logs is the only major change here. The rest is about refactoring the test suite.

@leonardBang is the support for binlog compression a blocker for these changes to get merged?

[...] needs more discussion and testing over this.

@yuxiqian do you have any specific discussion topics in mind? As for testing, I believe having the test suite pass would be a great start, although it's not clear why the CI didn't run the tests on this PR.

[morozov]

While this PR is being reviewed, I tried to test it and got a few questions.

[yuxiqian]

Thanks @morozov's kindly review!

@leonardBang is the support for binlog compression a blocker for these changes to get merged?

I did some experiment earlier, and noticed that binlog compression would break current binlog backfilling code. However, considering this feature was introduced in MySQL 8.0.20, perhaps it could be handled independently.

@yuxiqian do you have any specific discussion topics in mind? As for testing, I believe having the test suite pass would be a great start, although it's not clear why the CI didn't run the tests on this PR.

Seems there are a lot of merging conflicts blocking the CI. Will resolve them soon.

[JeffHu88]

Hi, could you please let me know when this issue will go live? Thanks!

[j4k4]

A release of this would be very helpful as MySQL 8.0 is nearing it's end of life.
E.g. AWS stops supporting version 8.0 soon.

[shengjk]

Hi, I would like to ask why we do not use Debezium directly, but implement it ourselves? Debezium already supports MySQL 8 and above, and Debezium updates much faster than we do.

[yuxiqian]

Hi, I would like to ask why we do not use Debezium directly, but implement it ourselves? Debezium already supports MySQL 8 and above, and Debezium updates much faster than we do.

Thanks for @shengjk's nice suggestion! For a long time we've been stuck at Debezium 1.9.x and monkey-patching Debezium, mainly for keeping compatibility with Java 8.

FLINK-36605 also suggests a similar idea, but it is more likely to happen after bumping to Flink 2.x and dropping Java 8 support.

[zhangshenghang]

@yuxiqian Hi, Is there any problem? Why is this PR in draft status? Does it have an expected merge date?

[yuxiqian]

@yuxiqian Hi, Is there any problem? Why is this PR in draft status? Does it have an expected merge date?

Thanks @zhangshenghang for tracing this issue!

IIRC, one concern from maintainers is we can't provide decent support for some newly introduced MySQL features, like Binlog compression, VECTOR data types, etc., without upgrading Debezium dependencies, which requires dropping Java 8 and Flink 1.x support. So this PR is, at best, a transitional solution.

The next feature release (3.6 or 4.0) has not been kicked-off, but there's some discussions on upgrading Debezium versions here (https://issues.apache.org/jira/browse/FLINK-36605), so I'm not sure if it's appropriate to introduce this temporary change at this time.

Anyway, I'll try to resolve all conflicts later this week so it could be marked ready for review.

[zhangshenghang]

@yuxiqian Hi, Is there any problem? Why is this PR in draft status? Does it have an expected merge date?

Thanks @zhangshenghang for tracing this issue!

IIRC, one concern from maintainers is we can't provide descent support for some newly introduced MySQL features, like Binlog compression, VECTOR data types, etc., without upgrading Debezium dependencies, which requires dropping Java 8 and Flink 1.x support. So this PR is, at best, a transitional solution.

The next feature release (3.6 or 4.0) has not been kicked-off, but there's some discussions on upgrading Debezium versions here (https://issues.apache.org/jira/browse/FLINK-36605), so I'm not sure if it's appropriate to introduce this temporary change at this time.

Anyway, I'll try to resolve all conflicts later this week so it could be marked ready for review.

@yuxiqian Thank you for your reply. Yes, upgrading to JDK 11 would be a significant change. If there could be a temporary transitional solution for 1.8, that would also be a good option.

[zhangshenghang]

@yuxiqian Hi, Is there any problem? Why is this PR in draft status? Does it have an expected merge date?

Thanks @zhangshenghang for tracing this issue!
IIRC, one concern from maintainers is we can't provide descent support for some newly introduced MySQL features, like Binlog compression, VECTOR data types, etc., without upgrading Debezium dependencies, which requires dropping Java 8 and Flink 1.x support. So this PR is, at best, a transitional solution.
The next feature release (3.6 or 4.0) has not been kicked-off, but there's some discussions on upgrading Debezium versions here (https://issues.apache.org/jira/browse/FLINK-36605), so I'm not sure if it's appropriate to introduce this temporary change at this time.
Anyway, I'll try to resolve all conflicts later this week so it could be marked ready for review.

@yuxiqian Thank you for your reply. Yes, upgrading to JDK 11 would be a significant change. If there could be a temporary transitional solution for 1.8, that would also be a good option.

@yuxiqian Hi，I saw a discussion here about upgrading Debezium to version 2.7.x: https://issues.apache.org/jira/browse/FLINK-36605.
However, it seems that version 2.7.x also does not support MySQL 8.4.
MySQL 8.4 supported starting from version 3.2

[yuxiqian]

Thanks @zhangshenghang for confirming this! Debezium 3.2 requiring Java 17 seems not very likely, since Flink 2.x still supports Java 11 (deprecated though). I'll fix CI test cases first.

[Romeoy]

Hi @yuxiqian , much appreciate! Is there any update for the plan to support mysql 8.4? As mysql 8.0 will not been matained soon for many cloud service providers, I believe many companies will met this issue as well. By the way, I saw another PR to resolve same issue #4196 .

[sridhard]

Hi,
When this will be released?

[cgnie]

Hope to see support for MySQL 8.x+ versions sooner.

[leonardBang]

Looks like many users/devs are waiting this feature, would you like to contribute this ?@yuxiqian

[yuxiqian]

Thanks everyone for the interest in this PR! I've addressed morozov and sjmudd's comments, added basic compatibility tests, and marked this ready for review.

I hope we can parameterize the entire test case with multiple MySQL versions for better coverage, but that might be suitable for another PR.

[Copilot]

Pull request overview

Updates the MySQL CDC connector and related integration tests to be compatible with MySQL 8.4+ where legacy terminology-based SQL commands (e.g., SHOW MASTER STATUS) are replaced.

Changes:

• Add MySQL 8.4 support by introducing a probed “show binary log status” statement and using it when reading binlog offsets.
• Add/adjust integration coverage to run compatibility tests across MySQL 5.7, 8.0, and 8.4.
• Update server-id conflict detection to recognize newer MySQL wording (“replica/source”).

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
pom.xml	Bumps JUnit version to support new parameterized class-based test style.
flink-cdc-e2e-tests/.../MySqlToIcebergE2eITCase.java	Removes redundant per-test MySQL container setup (now relies on shared test environment container).
.../testutils/MySqlVersion.java	Adds V8_4 enum entry for MySQL 8.4 test coverage.
.../table/MySqlCompatibilityITCase.java	Converts compatibility IT case to parameterized execution over MySQL versions including 8.4 and refactors container lifecycle/config generation.
.../debezium/DebeziumUtils.java	Switches binlog offset read to use the probed binary log status statement.
.../util/ErrorMessageUtils.java	Expands regex to match both old (slave/master) and new (replica/source) server-id conflict messages.
.../MySqlSnapshotChangeEventSource.java	Adds a Debezium-derived snapshot source implementation updated to use the probed binary log status statement.
.../MySqlConnection.java	Adds probing and caching for SHOW BINARY LOG STATUS vs SHOW MASTER STATUS to support MySQL 8.4+.
.../DebeziumSourceFunction.java	Extends server-id conflict detection to include the new MySQL “replica/source” phrasing.
.../MySqlPipelineCompatibilityITCase.java	Adds pipeline connector compatibility IT case parameterized across MySQL 5.7/8.0/8.4.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[leonardBang]

Thanks @yuxiqian for the update, current PR is in a good status, LGTM.
We can improve the multiple MySQL version tests later.