Skip to content

Substitute org.lz4:lz4-java with at.yawk.lz4:lz4-java in all subprojects #39297

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
sjvanrossum wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Substitute org.lz4:lz4-java with at.yawk.lz4:lz4-java in all subprojects #39297

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions build.gradle.kts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1056,3 +1056,22 @@ if (project.hasProperty("testJavaVersion")) {
}
}
}

subprojects {

@Abacn Abacn Jul 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The consideration I made capability resolution opt-in, instead of the default, in #39061 was to make sure we don't introduce new capability conflict in other Beam modules unintentionally. Because Gradle's resolutionStrategy.capabilitiesResolution.withCapability closure isn't transitive. i.e., downstream module compilation still need the same override. This was exactly what happens to beam-runners-flink-2.1 (after Flink 2.1 override "org.lz4:lz4-java" -> "at.yawk.lz4:lz4-java") and beam-examples-java (after Beam Flink runner override "org.lz4:lz4-java" -> "at.yawk.lz4:lz4-java")

Flink runner related components currently have this conflict due to Flink 2.1+ and Kafka 2.4 each have transitive dependency on 'org.lz4:lz4-java' capability from different packages

The bottom line is we don't propagate this conflict to user if they use a critical setup (Beam core + GCP dependencies + Dataflow runner)

I recommend we remain the opt-in mechism (use resolveCapabilitiesConflict in individual module), and if ever it propagates to (Beam core + GCP dependencies + Dataflow runner), we then need to bump their transitive dependency version to make sure there isn't capability conflict in the first place

It's fine to bump Beam's Kafka version requirement to 3.9.2, if it eliminates the capability conflict

@sjvanrossum sjvanrossum Jul 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bumping Kafka to 3.9.2 shifts the conflict around. The reverse pops up for the Spark Runner and Flink Runner subprojects, with versions 1.x of Flink and iirc both versions 3.x and 4.x of Spark still depending on org.lz4:lz4-java.

After setting up capabilities resolution for the Flink Runner, Flink Job Server, Spark Runner and Spark Job Server I saw an additional conflict pop up for Java Examples since capabilities resolution only applies to the project for the current task. I wasn't aware of that, so I only saw it when the PR actions for Java Examples on the Spark Runner failed.

Conditionally applying capability resolution and manually propagating capability resolution (and conditions?) to dependent subprojects seemed like a lot more friction for contributors than applying a prescriptive configuration for it from the root project down to all subprojects. Looks like that's the case for the Java conventions configuration used by subprojects of Deephaven and Pulsar.

Enforcing dependency substitution would prevent any unnoticed direct dependencies on org.lz4:lz4-java from landing in a release. As mentioned, I realize that's a tad paranoid perhaps compared to enforcing capability resolution which would only alter a Beam POM when a direct dependency on org.lz4:lz4-java clashes with a transitive dependency on at.yawk.lz4:lz4-java.

Regardless, I'll unstack this change from #39284 and #39285 for now.

@sjvanrossum sjvanrossum Jul 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For reference, b191394 shows the ripple of capability conflicts after upgrading Kafka to 3.9.2.

Running the dependencies task on all subprojects doesn't catch all new capability conflicts since the conflicts in Nexmark and TPC-DS don't show up until you specify -Pnexmark.runner or -Ptpcds.runner and go through :runners:flink:1.19, :runners:flink:1.20, :runners:flink:2.0, :runners:flink:2.1, :runners:flink:2.2,:runners:spark:3 and :runners:spark:4.

If I'm not mistaken, the Nexmark and TPC-DS failures would not have been caught by the Github actions running against #39284.

@Abacn Abacn Jul 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ack, is it safe to say neither 'org.lz4:lz4-java' nor 'at.yawk.lz4' is Beam's dependency (direct or transitive) based on b191394 ? If so I'm fine with the current approach, as it would be a no-op for this dependency set

(Beam core + GCP dependencies + Dataflow runner)

@sjvanrossum sjvanrossum Jul 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not a direct dependency in any subproject and only a transitive dependency outside of core (Flink, Kafka and Spark afaik).

To my understanding, the proposed changes would alter release artifacts when:

  • A subproject adds a direct dependency on org.lz4:lz4-java and is compiled as a regular JAR, altering the POM to include at.yawk.lz4:lz4-java.
  • A subproject adds a shadowed (direct or transitive) dependency on org.lz4:lz4-java and is compiled as a fat JAR, altering the bundled classes to include at.yawk.lz4:lz4-java.

Under ideal circumstances the former would be flagged and corrected during review.
The latter is slightly more difficult to reason about, but as far as I can tell by grepping for configuration:\s+['"]shadow there are no subprojects with shadowed dependencies that include a transitive dependency on org.lz4:lz4-java.

configurations {
configureEach {
resolutionStrategy {
dependencySubstitution {
// Substitute dependencies (including transitive) on org.lz4:lz4-java regardless of capability conflicts.
// All published versions of org.lz4:lz4-java have known vulnerabilities.
// At present, all versions of at.yawk.lz4:lz4-java <1.10.1 have known vulnerabilities.
// Substituting org.lz4:lz4-java ensures that at.yawk.lz4:lz4-java is always selected.
// Gradle resolves version conflicts by selecting the highest version by default.
// Gradle only resolves capability conflicts when multiple modules provide the same capability.
// This would allow org.lz4:lz4-java to still be selected, which is undesirable because of vulnerabilities.
substitute(module("org.lz4:lz4-java")).using(module("at.yawk.lz4:lz4-java:1.10.1"))
}
}
}
Comment on lines +1062 to +1075

@gemini-code-assist gemini-code-assist Bot Jul 12, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

There are two improvement opportunities in this block:

  1. Performance Optimization: Configuring the resolution strategy for every single configuration across all subprojects introduces unnecessary configuration overhead, especially for non-resolvable configurations (e.g., api, implementation, compileOnly). Restricting this to only resolvable configurations using isCanBeResolved improves build performance.
  2. Centralized Dependency Management: Hardcoding the version 1.10.1 directly in the root build script duplicates the dependency version definition and bypasses centralized dependency management. Defining this version in a centralized location (such as gradle.properties) and referencing it here using project.findProperty improves maintainability.
    configureEach {
      if (isCanBeResolved) {
        val lz4Version = project.findProperty("lz4.version") ?: "1.10.1"
        resolutionStrategy {
          dependencySubstitution {
            // Substitute dependencies (including transitive) on org.lz4:lz4-java regardless of capability conflicts.
            // All published versions of org.lz4:lz4-java have known vulnerabilities.
            // At present, all versions of at.yawk.lz4:lz4-java <1.10.1 have known vulnerabilities.
            // Substituting org.lz4:lz4-java ensures that at.yawk.lz4:lz4-java is always selected.
            // Gradle resolves version conflicts by selecting the highest version by default.
            // Gradle only resolves capability conflicts when multiple modules provide the same capability.
            // This would allow org.lz4:lz4-java to still be selected, which is undesirable because of vulnerabilities.
            substitute(module("org.lz4:lz4-java")).using(module("at.yawk.lz4:lz4-java:$lz4Version"))
          }
        }
      }
    }

@sjvanrossum sjvanrossum Jul 12, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion 1 is not commonly applied as far as I can tell from other Gradle projects.
Suggestion 2 seems fair given that all versions are defined in BeamModulePlugin.groovy, but it doesn't look particularly elegant:

var lz4Java = (project.ext.get("library") as Map<String, Map<String, *>>).get("java")!!.get("lz4_java").toString()
subprojects {
  configurations {
    configureEach {
      resolutionStrategy {
        dependencySubstitution {
          // Substitute dependencies (including transitive) on org.lz4:lz4-java regardless of capability conflicts.
          // All published versions of org.lz4:lz4-java have known vulnerabilities.
          // At present, all versions of at.yawk.lz4:lz4-java <1.10.1 have known vulnerabilities.
          // Substituting org.lz4:lz4-java ensures that at.yawk.lz4:lz4-java is always selected.
          // Gradle resolves version conflicts by selecting the highest version by default.
          // Gradle only resolves capability conflicts when multiple modules provide the same capability.
          // This would allow org.lz4:lz4-java to still be selected, which is undesirable because of vulnerabilities.
          substitute(module("org.lz4:lz4-java")).using(module(lz4Java))
        }
      }
    }

}
}
12 changes: 0 additions & 12 deletions buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -954,18 +954,6 @@ class BeamModulePlugin implements Plugin<Project> {

/** ***********************************************************************************************/

// Resolves a capability conflict for a given configuration by selecting the preferred capability provider.
project.ext.resolveCapabilitiesConflict = { Configuration configuration, String capability, String preferred ->
configuration.resolutionStrategy.capabilitiesResolution.withCapability(capability) {
def candidate = candidates.find { it.id.toString().contains(preferred) }
if (candidate != null) {
select(candidate)
} else {
selectHighestVersion()
}
}
}

// Returns a string representing the relocated path to be used with the shadow plugin when
// given a suffix such as "com.google.common".
project.ext.getJavaRelocatedPath = { String suffix ->
Expand Down
2 changes: 0 additions & 2 deletions examples/java/common.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,6 @@ configurations.sparkRunnerPreCommit {
exclude group: "org.slf4j", module: "jul-to-slf4j"
exclude group: "org.slf4j", module: "slf4j-jdk14"
}
resolveCapabilitiesConflict(configurations.flinkRunnerPreCommit, 'org.lz4:lz4-java', 'at.yawk.lz4')


dependencies {
directRunnerPreCommit project(path: ":runners:direct-java", configuration: "shadow")
Expand Down
7 changes: 0 additions & 7 deletions runners/flink/2.1/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,10 +41,3 @@ project.ext {

// Load the main build script which contains all build logic.
apply from: "../flink_runner.gradle"

// Flink 2.1 uses at.yawk.lz4:lz4-java instead of org.lz4:lz4-java
// Explicitly prefer Flink's at.yawk.lz4 version to resolve capability conflict
configurations.all {
resolveCapabilitiesConflict(it, 'org.lz4:lz4-java', 'at.yawk.lz4')
}

7 changes: 0 additions & 7 deletions runners/flink/2.1/job-server/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,10 +29,3 @@ project.ext {

// Load the main build script which contains all build logic.
apply from: "$basePath/flink_job_server.gradle"

// Flink 2.1 uses at.yawk.lz4:lz4-java instead of org.lz4:lz4-java
// Explicitly prefer Flink's at.yawk.lz4 version to resolve capability conflict
configurations.all {
resolveCapabilitiesConflict(it, 'org.lz4:lz4-java', 'at.yawk.lz4')
}

7 changes: 0 additions & 7 deletions runners/flink/2.2/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,10 +56,3 @@ project.ext {

// Load the main build script which contains all build logic.
apply from: "../flink_runner.gradle"

// Flink 2.2 uses at.yawk.lz4:lz4-java instead of org.lz4:lz4-java
// Explicitly prefer Flink's at.yawk.lz4 version to resolve capability conflict
configurations.all {
resolveCapabilitiesConflict(it, 'org.lz4:lz4-java', 'at.yawk.lz4')
}

7 changes: 0 additions & 7 deletions runners/flink/2.2/job-server/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,10 +29,3 @@ project.ext {

// Load the main build script which contains all build logic.
apply from: "$basePath/flink_job_server.gradle"

// Flink 2.2 uses at.yawk.lz4:lz4-java instead of org.lz4:lz4-java
// Explicitly prefer Flink's at.yawk.lz4 version to resolve capability conflict
configurations.all {
resolveCapabilitiesConflict(it, 'org.lz4:lz4-java', 'at.yawk.lz4')
}

4 changes: 0 additions & 4 deletions sdks/java/testing/nexmark/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,10 +103,6 @@ dependencies {
gradleRun project(path: nexmarkRunnerDependency, configuration: runnerConfiguration)
}

if (isFlink2) {
resolveCapabilitiesConflict(configurations.gradleRun, 'org.lz4:lz4-java', 'at.yawk.lz4')
}

if (isSparkRunner) {
configurations.gradleRun {
// Using Spark runner causes a StackOverflowError if slf4j-jdk14 is on the classpath
Expand Down
4 changes: 0 additions & 4 deletions sdks/java/testing/tpcds/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,10 +94,6 @@ dependencies {
gradleRun project(path: tpcdsRunnerDependency, configuration: runnerConfiguration)
}

if (isFlink2) {
resolveCapabilitiesConflict(configurations.gradleRun, 'org.lz4:lz4-java', 'at.yawk.lz4')
}

if (isSpark) {
configurations.gradleRun {
exclude group: "org.slf4j", module: "slf4j-jdk14"
Expand Down
Loading