[MINOR] Align supported Spark version references by LuciferYang · Pull Request #12525 · apache/gluten

LuciferYang · 2026-07-15T13:15:40Z

What changes were proposed in this pull request?

Spark 3.2 was dropped by #11351 / #11687 / #11731 / #11887; currently supported versions per README.md / docs/index.md / pom.xml profiles are Spark 3.3, 3.4, 3.5, 4.0, and 4.1. Various docs, comments, and small code paths still carry Spark 3.2 leftovers or predate the 4.0 / 4.1 additions. This PR aligns them.

Docs

docs/get-started/Velox.md: version table + prose updated to 3.3.1, 3.4.4, 3.5.5, 4.0.2, 4.1.1 (matching <spark.version> in each profile). Added a note that Spark 4.0/4.1 require JDK 17+ and Scala 2.13.
docs/velox-backend-limitations.md: dropped "For Spark3.2 and Spark3.3" from the "Override of Spark classes" section header (the mechanism applies to all supported versions); narrowed the two "Partition write" / "CTAS write" section headers from "For Spark3.2 and Spark3.3" to "For Spark3.3"; reworded the runtime warning to be version-agnostic; cleaned up grammar and a stray space in the reworded paragraph.
docs/get-started/VeloxQAT.md: replaced the hardcoded old version list with "all supported Spark versions" since the script iterates SUPPORTED_SPARK_VERSIONS.
docs/developers/HowToRelease.md: removed the stale spark-3.2.tar.gz example line.
tools/gluten-it/README.md: removed spark-3.2 from the profile list; kept spark-3.3, spark-3.4, spark-3.5. Spark 4.x is NOT advertised here yet because sbin/gluten-it.sh hardcodes SPARK_SCALA_VERSION=2.12, which conflicts with the Scala 2.13 required by the spark-4.0 / spark-4.1 profiles; advertising those profiles here would be misleading. Enabling Spark 4.x support for gluten-it is a separate change.
.github/ISSUE_TEMPLATE/bug.yml: dropdown updated to Spark-3.3.x through Spark-4.1.x (removed Spark-3.2.x, added Spark-4.1.x, normalized case).

Comments / small code

shims/spark33/.../OrcFileFormat.scala: rewrote the ScalaDoc header to remove the literal \n, fix grammar, and reference a real version-specific shim path (shims/spark34).
backends-velox/.../RowToVeloxColumnarExec.scala: removed // For spark 3.2. above withNewChildInternal; the override is the standard Spark 3.3+ API.
backends-velox/.../CudfNodeValidationRule.scala: replaced .find(_).isDefined (used because Spark 3.2 lacked TreeNode.exists) with .exists(_); dropped the stale comment.
backends-velox/.../VeloxAggregateFunctionsSuite.scala: dropped the stale "Spark 3.2 does not have this configuration" comment.
backends-velox/.../ArithmeticAnsiValidateSuite.scala: comment narrowed from "Spark 3.2 and 3.3" to "Spark 3.3".
tools/gluten-it/common/.../SparkJvmOptions.java: dropped the Spark-3.2 ClassNotFoundException fallback that returned ""; consolidated all reflection exceptions into a single multi-catch with a diagnostic message; routed Class.forName through the existing MODULE_OPTIONS_CLASS_NAME constant.

How was this patch tested?

mvn -pl backends-velox -am install -Pspark-3.5,backends-velox: SUCCESS
mvn -pl backends-velox scalastyle:check spotless:check -Pspark-3.5,backends-velox: SUCCESS
mvn -pl :spark-sql-columnar-shims-spark33 -am compile scalastyle:check spotless:check -Pspark-3.3: SUCCESS
javac SparkJvmOptions.java: SUCCESS
Docs render locally.

github-actions · 2026-07-15T13:16:09Z

Run Gluten Clickhouse CI on x86

Copilot

Pull request overview

This PR cleans up remaining Spark 3.2 references and updates docs/comments/code to reflect the currently supported Spark versions (3.3–4.1) across the Gluten project.

Changes:

Updates documentation and templates to remove Spark 3.2 references and include Spark 4.0/4.1 where applicable.
Simplifies small compatibility paths/comments that only existed for Spark 3.2 (e.g., TreeNode API usage, reflection fallback).
Adjusts several backend tests/comments to match Spark 3.3+ behavior expectations.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
tools/gluten-it/README.md	Updates supported `Spark-Version` profile list to 3.3–4.1.
tools/gluten-it/common/src/main/java/org/apache/gluten/integration/SparkJvmOptions.java	Removes Spark 3.2 fallback and consolidates reflection exception handling.
shims/spark33/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFileFormat.scala	Updates shim comment to remove Spark 3.2-specific wording.
docs/velox-backend-limitations.md	Narrows Spark 3.2/3.3 wording to Spark 3.3 in headers/warnings.
docs/get-started/VeloxQAT.md	Replaces hardcoded Spark version list with “all supported Spark versions”.
docs/get-started/Velox.md	Updates supported Spark versions table/prose to include 4.0/4.1 and remove 3.2.
docs/developers/HowToRelease.md	Removes stale Spark 3.2 tarball example line.
backends-velox/src/test/scala/org/apache/gluten/functions/ArithmeticAnsiValidateSuite.scala	Updates test comment about ANSI div-by-zero behavior to Spark 3.3.
backends-velox/src/test/scala/org/apache/gluten/execution/VeloxAggregateFunctionsSuite.scala	Removes stale Spark 3.2-only configuration comment.
backends-velox/src/main/scala/org/apache/gluten/extension/CudfNodeValidationRule.scala	Uses `TreeNode.exists` now that Spark 3.2 is no longer supported.
backends-velox/src/main/scala/org/apache/gluten/execution/RowToVeloxColumnarExec.scala	Removes Spark 3.2-only comment for `withNewChildInternal`.
.github/ISSUE_TEMPLATE/bug.yml	Updates Spark version dropdown to 3.3–4.1 and normalizes casing.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

LuciferYang · 2026-07-15T13:59:11Z

 must fall back to vanilla spark, etc.

-### Override of Spark classes (For Spark3.2 and Spark3.3)
+### Override of Spark classes (For Spark3.3)


Reworded the header to be generic ('Override of Spark classes') since the mechanism applies across all supported Spark versions. Applied in the latest force-push.

LuciferYang · 2026-07-15T13:59:13Z

 So you need to ensure preferentially load the Gluten jar to overwrite the jar of vanilla spark. Refer to [How to prioritize loading Gluten jars in Spark](https://github.com/apache/gluten/blob/main/docs/velox-backend-troubleshooting.md#incompatible-class-error-when-using-native-writer).

-If not officially supported spark3.2/3.3 version is used, NoSuchMethodError can be thrown at runtime. More details see [issue-4514](https://github.com/apache/gluten/issues/4514).
+If not officially supported spark3.3 version is used, NoSuchMethodError can be thrown at runtime. More details see [issue-4514](https://github.com/apache/gluten/issues/4514).


Reworded to 'If an unofficially supported Spark version is used' to avoid tying the warning to a specific Spark minor. Applied in the latest force-push.

LuciferYang · 2026-07-15T13:59:15Z


-### Override of Spark classes (For Spark3.2 and Spark3.3)
+### Override of Spark classes (For Spark3.3)
 Gluten avoids to modify Spark's existing code and use Spark APIs if possible. However, some APIs aren't exposed in Vanilla spark and we have to copy the Spark file and do the hardcode changes. The list of override classes can be found as ignoreClasses in package/pom.xml . If you use customized Spark, you may check if the files are modified in your spark, otherwise your changes will be overrided.


Fixed the typo 'overrided' -> 'overridden' since the line was already touched by this cleanup. Applied in the latest force-push.

LuciferYang · 2026-07-15T13:59:17Z

+ * This class is copied from Spark and modified for Gluten. \n
 * Gluten should make sure this class is loaded before the original class.
 * If new Spark releases accepts changes and make this class incompatible,
- * we can move this class to shims-spark32,
- * shims-spark33, etc.
+ * we can move this class to a specific shim (shims-spark33, etc.).


Rewrote the ScalaDoc to remove the literal '\n' and clean up the grammar ('If a new Spark release accepts changes that make this class incompatible ...'). Applied in the latest force-push.

LuciferYang · 2026-07-15T13:59:19Z

    try {
      final Class<?> clazz = Class.forName("org.apache.spark.launcher.JavaModuleOptions");
      final Method method = clazz.getMethod("defaultModuleOptions");
      return (String) method.invoke(null);
-    } catch (ClassNotFoundException e) {
-      // Could happen in Spark 3.2 which doesn't have this class yet.
-      return "";
-    } catch (NoSuchMethodException | InvocationTargetException | IllegalAccessException e) {
+    } catch (ClassNotFoundException


Now Class.forName uses the MODULE_OPTIONS_CLASS_NAME constant instead of duplicating the string literal. Applied in the latest force-push.

github-actions · 2026-07-15T13:59:43Z

Run Gluten Clickhouse CI on x86

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 5 comments.

LuciferYang · 2026-07-15T15:12:40Z

 ```

-Note: **Spark-Version** can only be **spark-3.2**, **spark-3.3**, **spark-3.4** or **spark-3.5**.
+Note: **Spark-Version** can only be **spark-3.3**, **spark-3.4**, **spark-3.5**, **spark-4.0** or **spark-4.1**.


Reverted the gluten-it profile list to spark-3.3 / 3.4 / 3.5 for now. sbin/gluten-it.sh currently hardcodes SPARK_SCALA_VERSION=2.12 and the wrapper script does not derive Scala 2.13 from the -Pspark-4.0 / -Pspark-4.1 profiles, so advertising those profiles here would be misleading. Making gluten-it actually work on Spark 4.x is a separate change and out of scope for this alignment PR.

LuciferYang · 2026-07-15T15:12:42Z

+        | NoSuchMethodException
+        | InvocationTargetException
+        | IllegalAccessException e) {
      throw new RuntimeException(e);


Added context to the RuntimeException: 'Failed to read Spark JVM module options via org.apache.spark.launcher.JavaModuleOptions#defaultModuleOptions'. Diagnosing classpath / Spark-version mismatches from sbin/gluten-it.sh should now be much easier.

LuciferYang · 2026-07-15T15:12:44Z

+/**
+ * This class is copied from Spark and modified for Gluten. Gluten should make sure this class is
+ * loaded before the original class. If a new Spark release accepts changes that make this class
+ * incompatible, we can move this class to a version-specific shim (shims-spark33, etc.).


Reworded the doc: 'we can move this file into a version-specific shim (e.g. shims/spark34) so it is only used by the Spark versions that need it.' This matches the actual layout (shims/spark33, shims/spark34, etc.) and the intent (dedicate a version-specific shim once a divergence appears).

LuciferYang · 2026-07-15T15:12:46Z

-### Override of Spark classes (For Spark3.2 and Spark3.3)
-Gluten avoids to modify Spark's existing code and use Spark APIs if possible. However, some APIs aren't exposed in Vanilla spark and we have to copy the Spark file and do the hardcode changes. The list of override classes can be found as ignoreClasses in package/pom.xml . If you use customized Spark, you may check if the files are modified in your spark, otherwise your changes will be overrided.
+### Override of Spark classes
+Gluten avoids to modify Spark's existing code and use Spark APIs if possible. However, some APIs aren't exposed in Vanilla spark and we have to copy the Spark file and do the hardcode changes. The list of override classes can be found as ignoreClasses in package/pom.xml . If you use customized Spark, you may check if the files are modified in your spark, otherwise your changes will be overridden.


Rewrote the paragraph for grammar and removed the stray space before the period: 'Gluten avoids modifying Spark's existing code and prefers Spark APIs when possible. However, some APIs are not exposed by vanilla Spark, so we have to copy the Spark file and apply hardcoded changes. The list of overridden classes can be found as `ignoreClasses` in `package/pom.xml`. If you use a customized Spark, check whether these files are modified in your Spark distribution, otherwise your changes will be overridden.'

LuciferYang · 2026-07-15T15:12:48Z

+| jdk   | openjdk8/jdk17                      |
+| scala | 2.12                                |


Added a note right below the version table: 'Spark 4.0 and 4.1 require JDK 17+ and Scala 2.13 (build with -Pspark-4.0 or -Pspark-4.1 plus -Pjava-17 -Pscala-2.13). Spark 3.3 to 3.5 can be built with JDK 8/17 and Scala 2.12.' This calls out the JDK/Scala requirement matrix without changing the table shape.

Spark 3.2 was dropped by apache#11351/apache#11687/apache#11731/apache#11887; currently supported versions per README.md / docs/index.md / pom.xml profiles are Spark 3.3, 3.4, 3.5, 4.0, and 4.1. Various docs, comments, and small code paths still carry Spark 3.2 leftovers or predate the 4.0/4.1 additions. This PR aligns them. Docs: - `docs/get-started/Velox.md`: version table + prose updated to 3.3.1, 3.4.4, 3.5.5, 4.0.2, 4.1.1 (matching `<spark.version>` in each profile). - `docs/velox-backend-limitations.md`: three section headers "For Spark3.2 and Spark3.3" reduced to "For Spark3.3"; the "spark3.2/3.3" runtime warning reduced to "spark3.3". - `docs/get-started/VeloxQAT.md`: replaced hardcoded old version list with "all supported Spark versions" since the script iterates `SUPPORTED_SPARK_VERSIONS`. - `docs/developers/HowToRelease.md`: removed the stale `spark-3.2.tar.gz` example line. - `tools/gluten-it/README.md`: profile list updated to spark-3.3, spark-3.4, spark-3.5, spark-4.0, spark-4.1. - `.github/ISSUE_TEMPLATE/bug.yml`: dropdown updated to Spark-3.3.x through Spark-4.1.x (removed Spark-3.2.x, added Spark-4.1.x, normalized case). Comments / small code: - `shims/spark33/.../OrcFileFormat.scala`: comment no longer references Spark 3.2 or a hypothetical shims-spark32. - `backends-velox/.../RowToVeloxColumnarExec.scala`: removed `// For spark 3.2.` above `withNewChildInternal`; the override is the standard Spark 3.3+ API. - `backends-velox/.../CudfNodeValidationRule.scala`: replaced `.find(_).isDefined` (used because Spark 3.2 lacked `TreeNode.exists`) with `.exists(_)`; dropped the stale comment. - `backends-velox/.../VeloxAggregateFunctionsSuite.scala`: dropped stale "Spark 3.2 does not have this configuration" comment. - `backends-velox/.../ArithmeticAnsiValidateSuite.scala`: comment narrowed from "Spark 3.2 and 3.3" to "Spark 3.3". - `tools/gluten-it/common/.../SparkJvmOptions.java`: dropped the Spark-3.2 `ClassNotFoundException` fallback that returned ""; consolidated all reflection exceptions into a single multi-catch.

github-actions · 2026-07-15T15:13:58Z

Run Gluten Clickhouse CI on x86

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 1 comment.

LuciferYang · 2026-07-15T15:24:33Z

 ```

-Note: **Spark-Version** can only be **spark-3.2**, **spark-3.3**, **spark-3.4** or **spark-3.5**.
+Note: **Spark-Version** can only be **spark-3.3**, **spark-3.4** or **spark-3.5**.


Updated the PR description to match the final scope: gluten-it/README.md keeps spark-3.3 / spark-3.4 / spark-3.5 and explicitly does NOT advertise spark-4.0 / spark-4.1 because sbin/gluten-it.sh currently hardcodes SPARK_SCALA_VERSION=2.12 which conflicts with the Scala 2.13 required by the Spark 4.x profiles. Making gluten-it work on Spark 4.x is a separate change out of scope for this alignment PR. The description now records this decision explicitly.

Copilot AI review requested due to automatic review settings July 15, 2026 13:15

github-actions Bot added CORE works for Gluten Core VELOX INFRA TOOLS DOCS labels Jul 15, 2026

Copilot started reviewing on behalf of LuciferYang July 15, 2026 13:16 View session

Copilot AI reviewed Jul 15, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 15, 2026 13:58

LuciferYang force-pushed the gluten-minor-align-spark-version-refs branch from 26eebcc to 2aa6178 Compare July 15, 2026 13:58

Copilot started reviewing on behalf of LuciferYang July 15, 2026 14:00 View session

Copilot AI reviewed Jul 15, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 15, 2026 15:12

LuciferYang force-pushed the gluten-minor-align-spark-version-refs branch from 2aa6178 to 6f83361 Compare July 15, 2026 15:12

Copilot started reviewing on behalf of LuciferYang July 15, 2026 15:14 View session

Copilot AI reviewed Jul 15, 2026

View reviewed changes

Uh oh!

Conversation

LuciferYang commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

github-actions Bot commented Jul 15, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 15, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 15, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

LuciferYang commented Jul 15, 2026 •

edited

Loading