[INFRA] Cache CI engine archives per URL with a reusable action by pan3793 · Pull Request #7561 · apache/kyuubi

pan3793 · 2026-07-13T08:13:36Z

Why are the changes needed?

The old cache-engine-archives cache is one blob under a constant key, so bumping any
engine invalidates all of them. This provisions engines explicitly instead:

New reusable action .github/actions/download-archive downloads a tarball and caches it
with actions/cache keyed by the URL (atomic download), so each engine caches
independently and stable versions restore instantly.
.github/scripts/resolve-engine-archive-urls.sh derives each URL from the Maven
properties (honoring the build's profiles/-D overrides); expose-engine-homes.sh
publishes SPARK_HOME/HIVE_HOME/FLINK_HOME from /opt.
Every master.yml job now resolves → downloads → exposes only the engines it needs, and
the build runs with -P{spark,flink,hive}-provided. The now-orphaned
cache-engine-archives action is removed.
Flink test helpers now honor FLINK_HOME first (fallback to the download dir for local
runs), matching Spark/Hive.
Spark extension modules gain a spark-home-from-archive profile, auto-activated only when
SPARK_HOME is unset, so the Standalone-cluster tests use the downloaded archive locally
but a provided SPARK_HOME is no longer overridden.
The hive-it avatica/CALCITE-1224 hack is now a single zip -d.

How was this patch tested?

Run by this PR's CI.

Locally checked: URL resolution across all matrix cases (Scala 2.13, Spark/Flink/Hive
version overrides, CDH6 empty-query), *_HOME resolution, the closer.lua redirect for
curl -fSL, the zip -d patch on a mock jar, and the extension SPARK_HOME profile via
the effective POM (inherits the env when set, falls back to the archive otherwise).

Was this patch authored or co-authored using generative AI tooling?

Assisted-by: Claude Opus 4.8

Its last consumer, the gluten workflow, was removed in KYUUBI apache#7562; the master.yml jobs now provision engines via the download-archive action. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

… tests Exposing SPARK_HOME into the CI environment disables the spark-home-from-archive profile (activated by !env.SPARK_HOME), which was the only place setting the scalatest environment variables. As a result SPARK_SCALA_VERSION was no longer set, so Spark local-cluster tests such as FinalStageResourceManagerSuite failed to launch executors against the downloaded binary distribution and hung until timeout. SPARK_SCALA_VERSION is always derivable from the build, so set it unconditionally in the base build; keep SPARK_HOME in the profile so it is only supplied when the environment does not already provide it. Maven merges the two environmentVariables maps, so both are present when the profile is active.

pan3793 · 2026-07-13T18:14:16Z

now the engine binary tarball takes less than 10s to download when hitting cache

wForget

Thanks @pan3793

wangzhigang1999

Nice work， thanks！

bowenliang123

LGTM. Thanks!

…le action ### Why are the changes needed? The old `cache-engine-archives` cache is one blob under a constant key, so bumping any engine invalidates all of them. This provisions engines explicitly instead: - New reusable action `.github/actions/download-archive` downloads a tarball and caches it with `actions/cache` **keyed by the URL** (atomic download), so each engine caches independently and stable versions restore instantly. - `.github/scripts/resolve-engine-archive-urls.sh` derives each URL from the Maven properties (honoring the build's profiles/`-D` overrides); `expose-engine-homes.sh` publishes `SPARK_HOME`/`HIVE_HOME`/`FLINK_HOME` from `/opt`. - Every `master.yml` job now resolves → downloads → exposes only the engines it needs, and the build runs with `-P{spark,flink,hive}-provided`. The now-orphaned `cache-engine-archives` action is removed. - Flink test helpers now honor `FLINK_HOME` first (fallback to the download dir for local runs), matching Spark/Hive. - Spark extension modules gain a `spark-home-from-archive` profile, auto-activated only when `SPARK_HOME` is unset, so the Standalone-cluster tests use the downloaded archive locally but a provided `SPARK_HOME` is no longer overridden. - The `hive-it` avatica/CALCITE-1224 hack is now a single `zip -d`. ### How was this patch tested? Run by this PR's CI. Locally checked: URL resolution across all matrix cases (Scala 2.13, Spark/Flink/Hive version overrides, CDH6 empty-query), `*_HOME` resolution, the `closer.lua` redirect for `curl -fSL`, the `zip -d` patch on a mock jar, and the extension `SPARK_HOME` profile via the effective POM (inherits the env when set, falls back to the archive otherwise). ### Was this patch authored or co-authored using generative AI tooling? Assisted-by: Claude Opus 4.8 Closes #7561 from pan3793/gha-dl. Closes #7561 1f9ad04 [Cheng Pan] [INFRA] Restore SPARK_SCALA_VERSION for Spark extension local-cluster tests 20a94b2 [Cheng Pan] [INFRA] Bump actions/cache to v6 in download-archive action a5c4208 [Cheng Pan] fix spark extension standalone tests 0010f4e [Cheng Pan] [INFRA] Remove orphaned cache-engine-archives action da90a49 [Cheng Pan] nit 13948d3 [Cheng Pan] [INFRA] Cache CI engine archives per URL with a reusable action Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> (cherry picked from commit 445a906) Signed-off-by: Cheng Pan <chengpan@apache.org>

pan3793 · 2026-07-14T02:22:32Z

thanks, merged to master/1.11

…eusable action The old `cache-engine-archives` cache is one blob under a constant key, so bumping any engine invalidates all of them. This provisions engines explicitly instead: - New reusable action `.github/actions/download-archive` downloads a tarball and caches it with `actions/cache` **keyed by the URL** (atomic download), so each engine caches independently and stable versions restore instantly. - `.github/scripts/resolve-engine-archive-urls.sh` derives each URL from the Maven properties (honoring the build's profiles/`-D` overrides); `expose-engine-homes.sh` publishes `SPARK_HOME`/`HIVE_HOME`/`FLINK_HOME` from `/opt`. - Every `master.yml` job now resolves → downloads → exposes only the engines it needs, and the build runs with `-P{spark,flink,hive}-provided`. The now-orphaned `cache-engine-archives` action is removed. - Flink test helpers now honor `FLINK_HOME` first (fallback to the download dir for local runs), matching Spark/Hive. - Spark extension modules gain a `spark-home-from-archive` profile, auto-activated only when `SPARK_HOME` is unset, so the Standalone-cluster tests use the downloaded archive locally but a provided `SPARK_HOME` is no longer overridden. - The `hive-it` avatica/CALCITE-1224 hack is now a single `zip -d`. Run by this PR's CI. Locally checked: URL resolution across all matrix cases (Scala 2.13, Spark/Flink/Hive version overrides, CDH6 empty-query), `*_HOME` resolution, the `closer.lua` redirect for `curl -fSL`, the `zip -d` patch on a mock jar, and the extension `SPARK_HOME` profile via the effective POM (inherits the env when set, falls back to the archive otherwise). Assisted-by: Claude Opus 4.8 Closes #7561 from pan3793/gha-dl. Closes #7561 1f9ad04 [Cheng Pan] [INFRA] Restore SPARK_SCALA_VERSION for Spark extension local-cluster tests 20a94b2 [Cheng Pan] [INFRA] Bump actions/cache to v6 in download-archive action a5c4208 [Cheng Pan] fix spark extension standalone tests 0010f4e [Cheng Pan] [INFRA] Remove orphaned cache-engine-archives action da90a49 [Cheng Pan] nit 13948d3 [Cheng Pan] [INFRA] Cache CI engine archives per URL with a reusable action Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org>

@v6

Backport of #7564 to branch-1.10. Combined with the parent backport of #7561 (2a278f2), this brings the GHA modernization (#7561 + #7564) to the 1.10 maintenance branch. Bring all GitHub Actions references on branch-1.10 to the latest, ASF-allowlist-compliant versions (matching master). The version normalization is reimplemented rather than cherry-picked, because master's #7564 delta (v7.0.0 -> v7, etc.) is anchored on versions branch-1.10 never carried; bumping branch-1.10's baseline directly is the conflict-free equivalent. Trusted actions (-> latest major tag): actions/checkout v4 -> v7 actions/setup-java v4 -> v5 actions/setup-python v5 -> v6 actions/setup-node v4 -> v6 actions/stale v9 -> v10 actions/upload-artifact v4 -> v7 actions/dependency-review-action v4 -> v5 actions/first-interaction v1 -> v3 actions/labeler v5 -> v6 (.github/labeler.yml already v6-schema) codecov/codecov-action v3 -> v7 actions/cache v4 -> v6 (in .github/actions/setup-maven; @v6 elsewhere unchanged) Docker actions (-> latest allowlist hash + tag comment): docker/build-push-action -> 53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7.3.0 docker/setup-buildx-action -> bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0 docker/setup-qemu-action -> 96fe6ef7f33517b61c61be40b68a1882f3264fb8 # v4.2.0 docker/login-action -> af1e73f918a031802d376d3c8bbc3fe56130a9b0 # v4.4.0 github/super-linter/slim@v7 was already latest; unchanged.

pan3793 · 2026-07-14T07:23:19Z

backported to branch-1.10 too

github-actions Bot added kind:infra license, community building, project builds, asf infra related, etc. module:flink module:integration-tests labels Jul 13, 2026

pan3793 and others added 4 commits July 13, 2026 17:50

[INFRA] Cache CI engine archives per URL with a reusable action

13948d3

nit

da90a49

[INFRA] Remove orphaned cache-engine-archives action

0010f4e

Its last consumer, the gluten workflow, was removed in KYUUBI apache#7562; the master.yml jobs now provision engines via the download-archive action. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

fix spark extension standalone tests

a5c4208

pan3793 force-pushed the gha-dl branch from 4814c41 to a5c4208 Compare July 13, 2026 11:39

github-actions Bot added module:spark kind:build module:extensions labels Jul 13, 2026

[INFRA] Bump actions/cache to v6 in download-archive action

20a94b2

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

pan3793 requested review from wForget and wangzhigang1999 July 13, 2026 11:53

pan3793 requested a review from bowenliang123 July 13, 2026 15:59

wForget approved these changes Jul 14, 2026

View reviewed changes

wangzhigang1999 approved these changes Jul 14, 2026

View reviewed changes

bowenliang123 approved these changes Jul 14, 2026

View reviewed changes

pan3793 closed this in 445a906 Jul 14, 2026

pan3793 self-assigned this Jul 14, 2026

pan3793 added this to the v1.11.2 milestone Jul 14, 2026

pan3793 deleted the gha-dl branch July 14, 2026 03:20

pan3793 mentioned this pull request Jul 14, 2026

[INFRA] Bump GHA workflows to latest action versions (branch-1.10 backport) #7566

Closed

pan3793 modified the milestones: v1.11.2, v1.10.4 Jul 14, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[INFRA] Cache CI engine archives per URL with a reusable action#7561

[INFRA] Cache CI engine archives per URL with a reusable action#7561
pan3793 wants to merge 6 commits into
apache:masterfrom
pan3793:gha-dl

pan3793 commented Jul 13, 2026 •

edited

Loading

Uh oh!

pan3793 commented Jul 13, 2026

Uh oh!

wForget left a comment

Uh oh!

wangzhigang1999 left a comment

Uh oh!

bowenliang123 left a comment

Uh oh!

pan3793 commented Jul 14, 2026

Uh oh!

pan3793 commented Jul 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

pan3793 commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are the changes needed?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

pan3793 commented Jul 13, 2026

Uh oh!

wForget left a comment

Choose a reason for hiding this comment

Uh oh!

wangzhigang1999 left a comment

Choose a reason for hiding this comment

Uh oh!

bowenliang123 left a comment

Choose a reason for hiding this comment

Uh oh!

pan3793 commented Jul 14, 2026

Uh oh!

pan3793 commented Jul 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

pan3793 commented Jul 13, 2026 •

edited

Loading