[ZEPPELIN-6311] Support JDK 17 - #5311
Draft
pan3793 wants to merge 4 commits into
Draft
Conversation
pan3793
force-pushed
the
jdk17
branch
3 times, most recently
from
August 10, 2026 09:15
c9d672a to
43f0295
Compare
Cassandra 3.x crashes on JDK 17 because Unsafe.objectFieldOffset() on hidden classes (lambdas) throws UnsupportedOperationException during PREPARE statement storage. This is a hard JVM restriction no flag can override. Cassandra 4.x handles this correctly. - Replace cassandra-unit with Testcontainers cassandra:4.1.3 - Rewrite CassandraInterpreterTest to use CassandraContainer - Update test expectation HTML files for Cassandra 4.x table options - Normalize localhost/<unresolved> InetSocketAddress format in assertions (JDK 17+ renders unresolved addresses differently) Assisted-by: GLM 5.2
Flink 1.x binary tgz bundles flink-scala_2.12 which shades scala-library, scala-compiler, and scala-reflect 2.12.7. Scala 2.12.7's JrtClassPath.asURLs() crashes on JDK 17+ with "/packages cannot be represented as URI" (scala/bug#11608, fixed in Scala 2.12.9). - Change flink.scala.version from 2.12.7 to 2.12.20 for all JDKs (binary incompatibility in Settings.usejavacp() between 2.12.7 and 2.12.20) - Add maven-antrun-plugin to flink-1.19/flink-1.20 profiles that strips scala/* from flink-scala_2.12-*.jar and copies 2.12.20 jars into FLINK_HOME/lib - Add patchFlinkScala() in DownloadUtils for integration tests that download Flink to ~/.cache/ (guards on scalaVersion=="2.12" and flinkVersion<2.0) Assisted-by: GLM 5.2
Add extraJavaTestArgs property to root pom.xml with --add-opens flags for JDK 17+ module access, and reference it from surefire argLine in all test modules. Remove the old per-module java-17 profile in spark/interpreter/pom.xml that duplicated these flags. Mirror the same flags in bin/common.sh and bin/common.cmd via JPMS_JAVA_OPTS so Zeppelin server and interpreter processes launched by bin scripts get the same module-access flags. Enable JDK 17 in the flink CI matrix. Assisted-by: GLM 5.2
Member
Author
|
@jongyoul, the current changes in this PR should make Zeppelin compatible with JDK 17, the real hacky part is the patch for Flink - Flink does not work out of the box until 2.0, let me know your thoughts about this part. (I may not be able to add support for Flink 2.0, and I hope others can take this if you think we should not hack Flink 1.19/1.20 but upgrade to Flink 2.0 first) |
Member
Author
|
I split the Cassandra part change into ZEPPELIN-6639 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is this PR for?
Enable Zeppelin to build, test, and run on JDK 17. This involves three independent areas of work:
What type of PR is it?
Improvement
What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-6311
How should this be tested?
CI passes on both JDK 11 and JDK 17 across all modules. Verified locally:
cassandramodule: 39 tests pass on JDK 17 with Dockerflink/flink-scala-2.12module: 7FlinkInterpreterTesttests pass on both JDK 11 and JDK 17bin/common.shsyntax-checked;JPMS_JAVA_OPTSverified inJAVA_OPTSandJAVA_INTP_OPTSQuestions
Details
1. Cassandra: Testcontainers migration
Cassandra 3.11.5 (via
cassandra-unit) crashes on JDK 17. DuringPREPAREstatement storage, Cassandra 3.x serializes lambda expressions. JDK 17 represents lambdas as hidden classes, andsun.misc.Unsafe.objectFieldOffset()on hidden classes throwsUnsupportedOperationException-- a hard JVM restriction that no flag can override.Changes:
cassandra-unitdependency withorg.testcontainers:cassandra(cassandra:4.1.3image)CassandraInterpreterTestto useCassandraContainerinstead ofEmbeddedCassandraServerHelperDescribeKeyspace_live_data.html,DescribeTable_live_data_complex_table.html,NoResultWithExecutionInfo.html) to match Cassandra 4.x table option output (e.g.additional_write_policy,read_repair = 'BLOCKING',speculative_retry = '99p',chunk_length_in_kb = 16)localhost/<unresolved>:portin test assertions (JDK 17+InetSocketAddress.toString()renders unresolved addresses differently)2. Flink: Scala 2.12.7 -> 2.12.20
Flink 1.19/1.20 binary tgz bundles
flink-scala_2.12-*.jarwhich shadesscala-library,scala-compiler, andscala-reflect2.12.7. Scala 2.12.7'sJrtClassPath.asURLs()crashes on JDK 17+ with:This was fixed in Scala 2.12.9 (scala/bug#11608).
Changes:
flink.scala.versionfrom2.12.7to2.12.20inflink/pom.xml(removes thejdk17+profile). The compile-time version must match the runtime version due to binary incompatibility inSettings.usejavacp()between 2.12.7 and 2.12.20.maven-antrun-plugintoflink-1.19andflink-1.20profiles that patches the downloaded Flink tgz: stripsscala/*classes fromflink-scala_2.12-*.jarviazip -d, then copiesscala-library/scala-compiler/scala-reflect2.12.20 jars intoFLINK_HOME/lib. Scoped to Flink 1.x only.patchFlinkScala()inDownloadUtilsfor integration tests that download Flink viaDownloadUtils.downloadFlink()to~/.cache/. Guards onscalaVersion == "2.12"andflinkVersion < 2.0.Experimental support for Java 17 was added in Flink 1.18. (FLINK-15736)
Flink 1.19/1.20 sticky with Scala 2.12.7 due to FLINK-12461, upgrade Scala is a necessary step for JDK 17 support.
3. JPMS args and JDK 17 CI
JDK 17 enforces strong module encapsulation. Many libraries (Netty, Mockito, etc.) require
--add-opensflags to access internal JDK APIs. These flags were already needed for surefire tests but were either missing or duplicated per-module.Assisted-by: GLM 5.2