[oadp-1.4] Make controller-gen/kustomize/golangci-lint/envtest tool-binary caching reliable - #2368
[oadp-1.4] Make controller-gen/kustomize/golangci-lint/envtest tool-binary caching reliable#2368kaovilai wants to merge 7 commits into
Conversation
Cherry-pick of the fix from oadp-dev. go-install-tool-branch only installs when the binary is missing, never verifying the pinned version against what's already on disk. Once a binary lands in bin/<branch>/, it's reused forever across branch switches and version bumps since bin/ is gitignored and nothing else resets it. kustomize and controller-gen's targets also weren't fully .PHONY (only the wrapper name was, not the binary path), so Make's own mtime-based staleness check could skip their recipe entirely before any version check ran. Introduce go-install-tool-versioned, which compares a sidecar <binary>.version marker file against the pinned version instead of introspecting the binary's own --version output. Binary introspection isn't reliable for every tool installed this way: kustomize's `version` command depends on ldflags its own release process sets, which `go install` doesn't set, so identically-installed kustomize binaries were observed reporting "(devel)", an unexpanded `$Format:%H$` git-archive placeholder, or a correct version string depending on unrelated build-time factors. oadp-1.4's golangci-lint target didn't have any version-check at all (unlike oadp-dev's, which had a partial/broken one), so it's folded into the same fix here. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository: openshift/coderabbit/.coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
Codecov Report❌ Patch coverage is Please upload reports for the commit d3ca37d to get more accurate results.
Additional details and impacted files@@ Coverage Diff @@
## oadp-1.4 #2368 +/- ##
============================================
+ Coverage 37.79% 38.47% +0.67%
============================================
Files 30 30
Lines 5033 5113 +80
============================================
+ Hits 1902 1967 +65
- Misses 2935 2949 +14
- Partials 196 197 +1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This branch's check-envtest-arch (landed via openshift#2100, Feb 2026) has been unconditionally reinstalling setup-envtest on every single `make test`/`make envtest` invocation since it merged: it treats any `$(ENVTEST) --help` failure as "wrong architecture," but setup-envtest's own --help exits 2 by its own convention even on a perfectly working binary. Since check-envtest-arch is itself .PHONY and a prerequisite of $(ENVTEST), that always-nonzero exit forces a delete+reinstall on every invocation, unconditionally, defeating the point of caching this binary at all. Verified by cross-compiling a real linux/amd64 setup-envtest and running it on this darwin/arm64 host: the shell reports exit code 126 specifically (POSIX "found but cannot execute" / exec format error) for a genuinely incompatible binary, distinct from the tool's own exit codes. Check that instead. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: kaovilai The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
1 similar comment
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: kaovilai The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Cherry-pick of the same extension from oadp-dev. The version-marker check alone isn't enough: a binary can have a correct .version marker but still be the wrong architecture, e.g. if a containerized build with a different GOARCH bind-mounts the host's bin/ directory (this Makefile documents exactly that workflow for `make test`: `docker run --platform linux/amd64 -v $PWD:$PWD ...`). Anything that does `go install` in there writes onto the host's real bin/ tree since it's the same mounted path, not a copy. Not envtest-specific — it can happen to any of these four cached tool binaries. go-install-tool-versioned now also probes `$(1) --version` and checks specifically for exit code 126 (POSIX "found but cannot execute" / exec format error), same technique as the envtest fix already on this branch. Verified controller-gen and kustomize (this branch's v4.5.5 pin) correctly detect and repair a wrong-arch binary even when its .version marker already matches the pinned version: cross-compiled a real linux/amd64 binary, copied it over the working native binary, confirmed each was detected and replaced. golangci-lint shares the identical code path but couldn't be exercised directly on this branch (its v1.54.2 pin fails to build against this sandbox's Go 1.26 toolchain, a pre-existing environment issue unrelated to this change). Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
Cherry-pick of the same fix from oadp-dev. An independent second-opinion
review caught a severe bug in the previous commit: the exit-126 probe
(`$(1) --version`, then a separate `if [ $? -eq 126 ]`) is a bare
command outside any &&/||/if guard. Under `set -e` — which this
Makefile's `.SHELLFLAGS = -ec` enables, honored by GNU Make 3.82+ — a
bare failing command aborts the whole recipe immediately, before the
exit-code check ever runs. My local macOS Make (3.81) silently ignores
.SHELLFLAGS, so none of my testing could have caught this.
Confirmed for real: installed GNU Make 4.4.1 via Homebrew and
reproduced the crash directly on this branch too — `make kustomize`
died with "Error 1" every time the binary already existed, because
kustomize's own `--version` exits 1 by its own convention even on a
perfectly healthy binary (this branch's kustomize v4.5.5 pin, same
issue as v5.2.1 on oadp-dev).
Replaced the whole exit-code-probe approach with `go version -m`,
which reads a binary's embedded module version and GOOS/GOARCH
directly from its build info, without executing it at all:
- No execution means no exit-code heuristic to get wrong, and no
-e hazard, for any of these four tools.
- No execution also closes a gap the review surfaced: exit-126
detection cannot work at all inside a container with
qemu-user-static/binfmt_misc registered (standard in multi-arch
CI/build images), since a wrong-arch binary just runs under
emulation and returns its own exit code instead of an
exec-format-error.
- Drops the sidecar `.version` marker file entirely — go version -m
reads the real, authoritative module version already embedded in
the binary.
This branch's check-envtest-arch (a separate target here, unlike
oadp-dev's inline check) is replaced outright by a plain call to the
same go-install-tool-versioned macro used by the other three tools.
Verified end-to-end with GNU Make 4.4.1: fresh install, idempotent
re-run (previously the exact crash case for kustomize/envtest), and
wrong-arch detection+repair, for controller-gen/kustomize/envtest.
golangci-lint@v1.54.2 still can't be exercised directly on this branch
(pre-existing build incompatibility with this sandbox's Go 1.26
toolchain, unrelated to this change — confirmed by testing whether a
newer v1.x pin would help: it does fix the build, but its updated
default linter set immediately surfaces ~20 pre-existing findings
across unrelated files, which is real scope growth this PR shouldn't
absorb; leaving the pin as-is). `make generate manifests bundle` zero
diff, `make test` fully green.
Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
ifeq evaluates its $(shell ...) condition at parse time, before any target's prerequisites run. On a cold bin/ (setup-envtest not yet installed), `$(ENVTEST) list` fails silently, the grep finds nothing, and ENVTESTPATH gets permanently redefined to force --arch=amd64 -- regardless of host arch -- even though setup-envtest is correctly installed for the host's native arch by the time the `test` recipe actually runs. Move the fallback entirely into the shell expression itself (still a recursively-expanded variable, so it's only evaluated when referenced in the `test` recipe, after `envtest` has installed the right arch). Fixes openshift#2377 Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
…in/" This reverts commit 999e0fb. Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
Both go-install-tool-versioned's doc comment and the envtest section's note narrated the debugging process that led here (earlier attempts, "confirmed with real GNU Make 4.4.1", "the previous exit-code-126 version... reliably crashed", a specific past PR reference) rather than just stating why the current code is shaped this way. Keeps the durable WHY (unreliable tool self-report, the set -e hazard as a standing fact about this Makefile, the qemu/binfmt_misc blind spot) as present-tense design rationale instead. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
|
@kaovilai: The following test failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
Cherry-pick of #2367 to
oadp-1.4— same fix, plus this branch's own pre-existing envtest arch-check bug (see below).Summary
go-install-tool-branchonly installs a build tool when its binary is missing, never verifying the pinned version against what's already on disk. Once a binary lands atbin/<branch>/<tool>, it's reused forever — even afterCONTROLLER_TOOLS_VERSION/KUSTOMIZE_VERSION/GOLANGCI_LINT_VERSIONchange — becausebin/is gitignored.kustomize/controller-gentargets also weren't fully.PHONY, so Make's own mtime-based staleness check could skip the recipe before any check ran.None of the four tools (
controller-gen/kustomize/golangci-lint/setup-envtest, the last folded in from #2152) were checked for architecture compatibility either. This repo's own Makefile documents adocker run --platform linux/amd64 -v $PWD:$PWD ...workflow to reproduce Prow's CI environment — that bind-mounts the host directory into a forced-arch container, so anygo installin there writes a foreign-arch binary straight onto the host'sbin/tree (same path, not a copy).This branch already had a bespoke
check-envtest-archtarget (from #2100) attempting to guard against exactly this forsetup-envtest, but it checked$(ENVTEST) --help's exit code and treated any nonzero result as "wrong architecture" —setup-envtest's own--helpexits2by its own convention even on a perfectly healthy binary, so it was unconditionally reinstallingsetup-envteston every single invocation.Design
go-install-tool-versionedreads a binary's embedded build info viago version -m— the module version andGOOS/GOARCHit was built with — instead of executing the binary or trusting a sidecar marker file:--version/--helpoutput isn't a reliable version or health signal: it can depend on ldflags a tool's own release process sets (whichgo installdoesn't set — e.g.kustomizecan report(devel)or an unexpanded$Format:%H$placeholder instead of its real version), and some tools exit nonzero on--versioneven when perfectly healthy (kustomizeexits1,setup-envtest's--helpexits2). A bare probe command that exits nonzero is also dangerous under this Makefile's.SHELLFLAGS = -ec(enablesset -e, honored by GNU Make 3.82+): it aborts the whole recipe unless wrapped in anif/||guard.qemu-user-static/binfmt_miscregistered (standard for multi-arch CI/build images) — the foreign-arch binary just runs under emulation and returns its own exit code, not an exec-format-error.Reading embedded build info sidesteps both: no execution means no exit-code heuristic to get wrong and no qemu blind spot.
setup-envtest's arch check is folded into this same shared macro rather than kept as a separate mechanism.Known limitations
ENVTESTPATH's arch selection has a separate, pre-existing bug — tracked as ENVTESTPATH arch-selection is decided at Makefile parse time, picks amd64 on any cold bin/ regardless of host arch #2377, not fixed here. Itsifeq-based amd64 fallback is decided at Makefile-parse time and picks the amd64-forced variant on any coldbin/, regardless of actual host arch — confirmed via direct repro (see ENVTESTPATH arch-selection is decided at Makefile parse time, picks amd64 on any cold bin/ regardless of host arch #2377). It's invisible on amd64 CI (no CI/release-branch risk) and only affects arm64 local dev with a cold checkout, so it's out of scope here rather than a reason to expand this PR's diff a third time.golangci-lint@v1.54.2(this branch's pin) doesn't build against a modern Go 1.26 toolchain — a pre-existing, unrelatedx/toolsincompatibility; CI pins its own Go version and isn't affected. Bumping to the latestv1.x(v1.64.8, avoidingv2's config/default-linter overhaul) does fix the build, but its updatedstaticcheck/ineffassign/gosimplelinters immediately surface ~20 pre-existing findings across unrelated files. Left the pin as-is — real scope growth this PR shouldn't absorb.bin/<tool>→ last-branch-wins, pre-existing behavior), a lower-priority observation, also worth a follow-up.Testing
Verified against GNU Make 4.4.1 specifically (installed via Homebrew), not just the macOS-default Make 3.81 — the latter silently ignores
.SHELLFLAGS, so it can't exerciseset -erecipe behavior at all.rm -rf bin/oadp-1.4/rm -f bin/setup-envtest, thengmake controller-gen/gmake kustomize/gmake envtestindividually — each installs fresh.linux/amd64) for controller-gen, kustomize, and envtest → each correctly detected viago version -m'sGOARCH/GOOSfields, removed, reinstalled with a valid native binary.gmake generate manifests bundle→ zero diff against a clean checkout.gmake test→ exit 0, all packages pass,Go code is formatted,api is up to date,bundle is up to date.Note
Responses generated with Claude