ci(mrbind): cache thirdparty/mrbind/build across all workflows via reusable action#6087
Open
Fedr wants to merge 6 commits into
Open
ci(mrbind): cache thirdparty/mrbind/build across all workflows via reusable action#6087Fedr wants to merge 6 commits into
Fedr wants to merge 6 commits into
Conversation
The Build MRBind step compiles a small CMake project (45 Ninja targets) that depends only on the pinned `thirdparty/mrbind` submodule and the MSYS2 clang-18 toolchain. Both are stable across most CI runs, but the build is re-done from scratch every time (~3 min wall-clock). Cache `thirdparty/mrbind/build` with a key derived from the submodule SHA, the MSYS2 lockfile, and the install script. On cache hit, skip the Build MRBind step entirely; the existing mrbind.exe / mrbind_gen_c.exe / mrbind_gen_csharp.exe are consumed by the subsequent "Generate C bindings" / "Generate C# bindings" steps as before. Same pattern as the existing msys2_packages cache in .github/actions/install-msys2-mrbind. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fedr
added
disable-build-macos
disable-build-emscripten
disable-build-ubuntu-x64
disable-build-ubuntu-arm64
disable-build-linux-vcpkg
full-ci
Apply the cache-on-submodule-SHA pattern from the previous commit to
every workflow that builds MRBind:
- build-test-windows.yml (already done; refactored to call the
action)
- build-test-ubuntu-x64.yml
- build-test-ubuntu-arm64.yml
- build-test-macos.yml (split deps install from the build)
- build-test-linux-vcpkg.yml
- pip-build.yml (3 sections: rocky, windows, macos)
- update-docs-manual.yml
The composite action at .github/actions/build-mrbind:
- computes the `thirdparty/mrbind` submodule SHA,
- caches `thirdparty/mrbind/build` with a key derived from the
caller-supplied prefix (platform/toolchain), the submodule SHA, the
hash of the build script, and a caller-supplied extra key,
- runs the build script only on cache miss.
The cache-key-prefix includes the Docker image tag for
container-based jobs (ubuntu, rocky), so the cache invalidates when the
toolchain image changes. On macOS / Windows we use the clang version
file or the MSYS2 lockfile as the toolchain identifier.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fedr
changed the title
The arm64 matrix in build-test-macos.yml has two ARM runners with different homebrew prefixes: matrix.os=github-arm -> macos-latest -> /opt/homebrew matrix.os=arm -> self-hosted ARM -> /Users/runner/.homebrew `runner.arch` is ARM64 on both, so they were sharing a cache entry. The cached mrbind binary embeds the homebrew prefix in its rpath, so restoring a binary built on one runner onto the other fails with `dyld: Library not loaded: /Users/runner/.homebrew/opt/llvm@18/lib/libLLVM.dylib`. Use matrix.os instead. In pip-build.yml's macOS section, use matrix.instance for the same reason. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Both build-test-linux-vcpkg.yml and pip-build.yml's rocky section build mrbind inside the same `meshlib/meshlib-rockylinux8-vcpkg-<arch>:<tag>` Docker image using the same `install_mrbind_rockylinux.sh` script. The resulting binaries are identical, but they were using distinct cache key prefixes (`rocky-...` vs `rocky-pip-...`), so each workflow built its own copy. Unify both prefixes to `rockylinux8-vcpkg-<arch>-<tag>` -- matching the Docker image name exactly -- so the two callers share one cache entry. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… pip
Two changes that together let build-test-macos.yml and pip-build.yml
share a single mrbind cache entry per runner image:
1. Pin the arm64 Debug job's runner from `macos-latest` to `macos-14`
so it matches pip-build.yml's pinned `macos-14` runner. The
`latest` label is rolling; if it advances to macos-15-arm while
pip-build still pins macos-14, a binary cached under one would be
restored on the other (and may break, as happened earlier with the
self-hosted runner's homebrew prefix).
2. Add `instance:` to each matrix entry in build-test-macos.yml,
matching the field name pip-build.yml already uses, and key the
cache as `${{ matrix.instance }}-clang` in both. With the pin
above, the two workflows now resolve to identical keys for the two
GitHub-hosted images:
macos-15-intel-clang-<...> (build-test x64 + pip x86)
macos-14-clang-<...> (build-test arm64 Debug + pip arm64)
self-hosted-arm-clang-<...> (build-test arm64 Release, alone)
Saves one mrbind build per CI run and one cache entry (~65 MiB).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
.github/actions/build-mrbindcachesthirdparty/mrbind/buildkeyed on the submodule SHA + build script hash + a caller-supplied platform/toolchain key.build-test-windows.ymlbuild-test-ubuntu-x64.ymlbuild-test-ubuntu-arm64.ymlbuild-test-macos.ymlbuild-test-linux-vcpkg.ymlpip-build.yml(rocky, windows, macos)update-docs-manual.ymlWhy
In a typical Windows run (e.g. job 75535849635) Build MRBind takes ~3:07:
[1/45]at09:20:11→[45/45]at09:23:18. Compilation is already parallel (Ninja, all cores) — the wall-clock is bounded by a handful of heavy translation units, not lack of parallelism. The submodule is pinned (a238f863…) and the toolchain is pinned per platform, so re-doing the build every run is pure waste.Cache key composition
The composite action builds the key from:
cache-key-prefix— caller-supplied platform/toolchain identifier.thirdparty/mrbindsubmodule SHA (git -C thirdparty/mrbind rev-parse HEAD).extra-cache-key— caller-supplied (ahashFiles(…)over the relevant compiler manifest).Per-platform identifiers used:
windows-clang18msys2_package_hashes_clang18.txtubuntu-<os>-<docker_image_tag>clang_version.txtubuntu-arm64-<os>-<docker_image_tag>clang_version.txtmacos[-pip]-<runner.arch>-clangclang_version.txtrocky-<arch>-<docker_image_tag>rocky-pip-<os>-<vcpkg_docker_image_tag>ubuntu-docs-clangclang_version.txtNotes
install_mrbind_*.{sh,bat}scriptsrm -rf build/rmdir /S /Q buildat the start. Because the entire Build MRBind step is gated off viaif: cache-hit != 'true'(inside the action), that destructive line never runs on cache hit and the restored build dir is preserved.install_deps_macos.sh(homebrew deps) withinstall_mrbind_macos.sh(the build). Split into two steps so caching applies only to the build; deps always install.pip-build.ymlmacOS section was likewise split, withbuild_thirdparty.sh+install_deps_macos.shkept in the "Install thirdparty libs" step and the MRBind build extracted.Test plan
thirdparty/mrbindand confirm cache miss → rebuild on the next CI run.scripts/mrbind/clang_version.txt(or the msys2 lockfile) and confirm the same.