Skip to content

[AURON #2407] [BUILD] Fix the invalid Spark 4.1 Docker build example.#2408

Open
slfan1989 wants to merge 1 commit into
apache:masterfrom
slfan1989:auron-2407
Open

[AURON #2407] [BUILD] Fix the invalid Spark 4.1 Docker build example.#2408
slfan1989 wants to merge 1 commit into
apache:masterfrom
slfan1989:auron-2407

Conversation

@slfan1989

@slfan1989 slfan1989 commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #2407

Rationale for this change

The Docker build example generated by auron-build.sh --help used Spark 4.1 together with the centos7 image.

The centos7 image only provides JDK 8, while Spark 4.x requires JDK 17 or later. The example also enabled Iceberg and Hudi for Spark versions with which those integrations are not compatible.

As a result, users following the documented example encountered Java version or integration compatibility errors before completing the build.

What changes are included in this PR?

  • Replace the single Docker build example with explicit examples for
    Spark 3.5, Spark 4.0, and Spark 4.1.
  • Use the rockylinux8 image for these examples because it supports the
    JDK selection required by Spark 4.x.
  • Use Scala 2.12 for Spark 3.5 and Scala 2.13 for Spark 4.x.
  • Include only compatible third-party integrations in each example:
    • Spark 3.5 includes Iceberg 1.10.1 and Hudi 0.15.
    • Spark 4.0 includes Iceberg 1.10.1 but excludes Hudi.
    • Spark 4.1 excludes both Iceberg and Hudi.
  • Add a build compatibility matrix to CONTRIBUTING.md.
  • Document the Spark, Scala, JDK, Docker image, Flink, Celeborn,
    Uniffle, Paimon, Iceberg, and Hudi combinations.
  • Update the existing Docker build example in CONTRIBUTING.md to use
    rockylinux8.
  • Document that the compatibility matrix and Shell help examples should
    be updated together.

Are there any user-facing changes?

Yes.

The help output now provides separate Docker build examples for Spark
3.5, Spark 4.0, and Spark 4.1. Each example uses an appropriate Scala,
JDK, Docker image, and third-party integration combination.

Contributors can also consult the compatibility matrix in
CONTRIBUTING.md before selecting build arguments.

How was this patch tested?

./auron-build.sh --help

Usage: ./auron-build.sh [OPTIONS] <maven build options>
Build Auron project with specified Maven profiles

Options:
  --pre                    Activate pre-release profile
  --release                Activate release profile
  --clean <true|false>     Clean before build (default: true)
  --skiptests <true|false> Skip unit tests (default: true)
  --sparktests <true|false> Run spark tests (default: false)
  --docker <true|false>    Build in Docker environment (default: false)
  --threads <N|NC>         Maven build threads (e.g. 1, 4, 1C). Default: local unset, docker 8
  --image <NAME>           Docker image to use (e.g. centos7,ubuntu24,rockylinux8,debian11,azurelinux3, default: centos7)
  --sparkver <VERSION>     Specify Spark version (e.g. 3.0,3.1,3.2,3.3,3.4,3.5,4.0,4.1)
  --flinkver <VERSION>     Specify Flink version (e.g. 1.18)
  --scalaver <VERSION>     Specify Scala version (e.g. 2.12,2.13)
  --celeborn <VERSION>     Specify Celeborn version (e.g. 0.5,0.6)
  --uniffle <VERSION>      Specify Uniffle version (e.g. 0.10)
  --paimon <VERSION>       Specify Paimon version (e.g. 1.2)
  --iceberg <VERSION>      Specify Iceberg version (e.g. 1.10.1)
  --hudi <VERSION>         Specify Hudi version (e.g. 0.15)
  -h, --help               Show this help message

Examples:
  ./auron-build.sh --pre --sparkver 4.1 --scalaver 2.13 -DskipBuildNative
  # Spark 3.5 with all supported third-party integrations
  ./auron-build.sh --docker true --image rockylinux8 --clean true --skiptests true --release --sparkver 3.5 --scalaver 2.12 --flinkver 1.18 --celeborn 0.6 --uniffle 0.10 --paimon 1.2 --iceberg 1.10.1 --hudi 0.15
  # Spark 4.0 (Hudi is not supported)
  ./auron-build.sh --docker true --image rockylinux8 --clean true --skiptests true --release --sparkver 4.0 --scalaver 2.13 --flinkver 1.18 --celeborn 0.6 --uniffle 0.10 --paimon 1.2 --iceberg 1.10.1
  # Spark 4.1 (Iceberg and Hudi are not supported)
  ./auron-build.sh --docker true --image rockylinux8 --clean true --skiptests true --release --sparkver 4.1 --scalaver 2.13 --flinkver 1.18 --celeborn 0.6 --uniffle 0.10 --paimon 1.2

Was this patch authored or co-authored using generative AI tooling?

  • Yes
  • No

If yes, include: Generated-by: <tool name and version>

ASF guidance: https://www.apache.org/legal/generative-tooling.html

…ample.

Signed-off-by: slfan1989 <slfan1989@apache.org>
Copilot AI review requested due to automatic review settings July 21, 2026 00:14
@github-actions github-actions Bot added documentation Improvements or additions to documentation build labels Jul 21, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates the build documentation and auron-build.sh --help output to provide valid, Spark-version-specific Docker build examples (notably avoiding Spark 4.x with centos7/JDK 8 and excluding incompatible integrations), and adds a documented compatibility matrix for contributors.

Changes:

  • Update CONTRIBUTING.md Docker example to use rockylinux8 and add a build compatibility matrix covering Spark/Scala/JDK/image and integration combinations.
  • Update auron-build.sh --help to print separate Docker build examples for Spark 3.5, 4.0, and 4.1 with compatible Scala/JDK/image/integration selections.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
CONTRIBUTING.md Updates Docker build example and adds a build compatibility matrix documenting supported combinations.
auron-build.sh Replaces the single Docker example in --help with explicit, Spark-version-specific Docker build examples.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread auron-build.sh
Comment on lines 83 to 84
"--iceberg ${SUPPORTED_ICEBERG_VERSIONS[*]: -1}" \
"--hudi ${SUPPORTED_HUDI_VERSIONS[*]: -1}"
Comment thread auron-build.sh
"--celeborn ${SUPPORTED_CELEBORN_VERSIONS[*]: -1}" \
"--uniffle ${SUPPORTED_UNIFFLE_VERSIONS[*]: -1}" \
"--paimon ${SUPPORTED_PAIMON_VERSIONS[*]: -1}" \
"--iceberg ${SUPPORTED_ICEBERG_VERSIONS[*]: -1}"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUILD] Fix the invalid Spark 4.1 Docker build example

2 participants