[INFRA] Bump GHA workflows to latest action versions (branch-1.10 backport)#7566
Closed
pan3793 wants to merge 3 commits into
Closed
[INFRA] Bump GHA workflows to latest action versions (branch-1.10 backport)#7566pan3793 wants to merge 3 commits into
pan3793 wants to merge 3 commits into
Conversation
…reusable action
The old `cache-engine-archives` cache is one blob under a constant key, so bumping any
engine invalidates all of them. This provisions engines explicitly instead:
- New reusable action `.github/actions/download-archive` downloads a tarball and caches it
with `actions/cache` **keyed by the URL** (atomic download), so each engine caches
independently and stable versions restore instantly.
- `.github/scripts/resolve-engine-archive-urls.sh` derives each URL from the Maven
properties (honoring the build's profiles/`-D` overrides); `expose-engine-homes.sh`
publishes `SPARK_HOME`/`HIVE_HOME`/`FLINK_HOME` from `/opt`.
- Every `master.yml` job now resolves → downloads → exposes only the engines it needs, and
the build runs with `-P{spark,flink,hive}-provided`. The now-orphaned
`cache-engine-archives` action is removed.
- Flink test helpers now honor `FLINK_HOME` first (fallback to the download dir for local
runs), matching Spark/Hive.
- Spark extension modules gain a `spark-home-from-archive` profile, auto-activated only when
`SPARK_HOME` is unset, so the Standalone-cluster tests use the downloaded archive locally
but a provided `SPARK_HOME` is no longer overridden.
- The `hive-it` avatica/CALCITE-1224 hack is now a single `zip -d`.
Run by this PR's CI.
Locally checked: URL resolution across all matrix cases (Scala 2.13, Spark/Flink/Hive
version overrides, CDH6 empty-query), `*_HOME` resolution, the `closer.lua` redirect for
`curl -fSL`, the `zip -d` patch on a mock jar, and the extension `SPARK_HOME` profile via
the effective POM (inherits the env when set, falls back to the archive otherwise).
Assisted-by: Claude Opus 4.8
Closes apache#7561 from pan3793/gha-dl.
Closes apache#7561
1f9ad04 [Cheng Pan] [INFRA] Restore SPARK_SCALA_VERSION for Spark extension local-cluster tests
20a94b2 [Cheng Pan] [INFRA] Bump actions/cache to v6 in download-archive action
a5c4208 [Cheng Pan] fix spark extension standalone tests
0010f4e [Cheng Pan] [INFRA] Remove orphaned cache-engine-archives action
da90a49 [Cheng Pan] nit
13948d3 [Cheng Pan] [INFRA] Cache CI engine archives per URL with a reusable action
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
Backport of apache#7564 to branch-1.10. Combined with the parent backport of apache#7561 (2a278f2), this brings the GHA modernization (apache#7561 + apache#7564) to the 1.10 maintenance branch. Bring all GitHub Actions references on branch-1.10 to the latest, ASF-allowlist-compliant versions (matching master). The version normalization is reimplemented rather than cherry-picked, because master's apache#7564 delta (v7.0.0 -> v7, etc.) is anchored on versions branch-1.10 never carried; bumping branch-1.10's baseline directly is the conflict-free equivalent. Trusted actions (-> latest major tag): actions/checkout v4 -> v7 actions/setup-java v4 -> v5 actions/setup-python v5 -> v6 actions/setup-node v4 -> v6 actions/stale v9 -> v10 actions/upload-artifact v4 -> v7 actions/dependency-review-action v4 -> v5 actions/first-interaction v1 -> v3 actions/labeler v5 -> v6 (.github/labeler.yml already v6-schema) codecov/codecov-action v3 -> v7 actions/cache v4 -> v6 (in .github/actions/setup-maven; @v6 elsewhere unchanged) Docker actions (-> latest allowlist hash + tag comment): docker/build-push-action -> 53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7.3.0 docker/setup-buildx-action -> bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0 docker/setup-qemu-action -> 96fe6ef7f33517b61c61be40b68a1882f3264fb8 # v4.2.0 docker/login-action -> af1e73f918a031802d376d3c8bbc3fe56130a9b0 # v4.4.0 github/super-linter/slim@v7 was already latest; unchanged.
This removes the kyuubi-gluten-it module, the gluten-it Maven profile, the CI workflow, and the GlutenSuiteMixin test utility. Gluten IT is not maintained well, let's just remove it. Pass GHA. Assisted-by: GLM 5.2 Closes apache#7562 from pan3793/remove-gluten-it. Closes apache#7562 45918d4 [Cheng Pan] [INFRA] Remove gluten integration test module Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## branch-1.10 #7566 +/- ##
===========================================
Coverage 0.00% 0.00%
===========================================
Files 692 692
Lines 43028 43028
Branches 5849 5849
===========================================
Misses 43028 43028 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Logical backport of the GHA modernization to
branch-1.10, delivered as three commits mirroring master's split (#7561→#7564→#7562).Commits
2a278f2db—[KYUUBI #7561] Cache CI engine archives per URL with a reusable action(cherry-pick of445a906c2). Replaces the single-blobcache-engine-archivesaction with per-URLdownload-archivecaching, addsresolve-engine-archive-urls.sh/expose-engine-homes.sh, rewires everymaster.ymljob to resolve → download → expose only the engines it needs, adds thespark-home-from-archiveprofile to the Spark-3.3/3.4/3.5 extension poms, and makes the Flink test helpers honorFLINK_HOME.defaultjob'sSetup Pythonstep — master switchedpython-version: '3.9'→${{ matrix.python }}(master has apythonmatrix dim), butbranch-1.10'sdefaultmatrix has none, so the value is kept as'3.9'while [INFRA] Cache CI engine archives per URL with a reusable action #7561's resolve/download/expose steps are adopted verbatim. All other jobs auto-merged.kyuubi-extension-spark-4-0/-4-1poms [INFRA] Cache CI engine archives per URL with a reusable action #7561 modified are absent onbranch-1.10→ dropped (modify/delete).62d4b2b23—[INFRA] Bump GHA workflows to latest action versions(logical backport of99bed3885/ [INFRA] Bump GHA workflows to latest action versions #7564). Master's [INFRA] Bump GHA workflows to latest action versions #7564 delta is anchored on versionsbranch-1.10never carried, so it is reimplemented by bumpingbranch-1.10's baseline directly — the conflict-free equivalent. The resultinguses:set now matches master exactly (latest major tags for trusted actions; latest allowlist hashes +# vX.Y.Zcomments for the Docker actions).69b77b1cd—[KYUUBI #7562] [INFRA] Remove gluten integration test module(cherry-pick of00ba4e65d). This is needed because [INFRA] Cache CI engine archives per URL with a reusable action #7561 deletescache-engine-archivesbut, unlike master,branch-1.10still hadgluten.ymlreferencing it (on master the gluten workflow had been removed earlier by [INFRA] Remove gluten integration test module #7562, so [INFRA] Cache CI engine archives per URL with a reusable action #7561 never converted it). Removing the gluten IT module + workflow here eliminates that dangling reference cleanly. The 6 modify/delete conflicts (gluten.yml,gluten.md,GlutenSuiteMixin.scala, and the diverged gluten-IT files) are resolved to deletion;integration-tests/pom.xmlanddocs/.../index.rstauto-merged.How was this patch tested?
git grep gluten→ none) and has nocache-engine-archivesreferences;gluten.ymlis gone.uses:set onbranch-1.10matches master exactly; no old action-version tokens remain../dev/reformatwas run; it only failed on the Spotlessblackstep (a local pyenv-shim issue, exit 127) and modified no files — YAML is not Spotless-managed, and the [INFRA] Cache CI engine archives per URL with a reusable action #7561 source changes are authored against master and Spotless-clean.master.ymlmatrix.Was this patch authored or co-authored using generative AI tooling?
Assisted-by: codemaker with glm-5.2