Skip to content

[oadp-1.4] Make controller-gen/kustomize/golangci-lint/envtest tool-binary caching reliable - #2368

Open
kaovilai wants to merge 7 commits into
openshift:oadp-1.4from
kaovilai:fix-tool-version-check-oadp-1.4
Open

[oadp-1.4] Make controller-gen/kustomize/golangci-lint/envtest tool-binary caching reliable#2368
kaovilai wants to merge 7 commits into
openshift:oadp-1.4from
kaovilai:fix-tool-version-check-oadp-1.4

Conversation

@kaovilai

@kaovilai kaovilai commented Aug 10, 2026

Copy link
Copy Markdown
Member

Cherry-pick of #2367 to oadp-1.4 — same fix, plus this branch's own pre-existing envtest arch-check bug (see below).

Summary

go-install-tool-branch only installs a build tool when its binary is missing, never verifying the pinned version against what's already on disk. Once a binary lands at bin/<branch>/<tool>, it's reused forever — even after CONTROLLER_TOOLS_VERSION/KUSTOMIZE_VERSION/GOLANGCI_LINT_VERSION change — because bin/ is gitignored. kustomize/controller-gen targets also weren't fully .PHONY, so Make's own mtime-based staleness check could skip the recipe before any check ran.

None of the four tools (controller-gen/kustomize/golangci-lint/setup-envtest, the last folded in from #2152) were checked for architecture compatibility either. This repo's own Makefile documents a docker run --platform linux/amd64 -v $PWD:$PWD ... workflow to reproduce Prow's CI environment — that bind-mounts the host directory into a forced-arch container, so any go install in there writes a foreign-arch binary straight onto the host's bin/ tree (same path, not a copy).

This branch already had a bespoke check-envtest-arch target (from #2100) attempting to guard against exactly this for setup-envtest, but it checked $(ENVTEST) --help's exit code and treated any nonzero result as "wrong architecture" — setup-envtest's own --help exits 2 by its own convention even on a perfectly healthy binary, so it was unconditionally reinstalling setup-envtest on every single invocation.

Design

go-install-tool-versioned reads a binary's embedded build info via go version -m — the module version and GOOS/GOARCH it was built with — instead of executing the binary or trusting a sidecar marker file:

  • A binary's own --version/--help output isn't a reliable version or health signal: it can depend on ldflags a tool's own release process sets (which go install doesn't set — e.g. kustomize can report (devel) or an unexpanded $Format:%H$ placeholder instead of its real version), and some tools exit nonzero on --version even when perfectly healthy (kustomize exits 1, setup-envtest's --help exits 2). A bare probe command that exits nonzero is also dangerous under this Makefile's .SHELLFLAGS = -ec (enables set -e, honored by GNU Make 3.82+): it aborts the whole recipe unless wrapped in an if/|| guard.
  • Executing the binary to test compatibility can't detect a wrong-arch binary at all inside a container with qemu-user-static/binfmt_misc registered (standard for multi-arch CI/build images) — the foreign-arch binary just runs under emulation and returns its own exit code, not an exec-format-error.

Reading embedded build info sidesteps both: no execution means no exit-code heuristic to get wrong and no qemu blind spot. setup-envtest's arch check is folded into this same shared macro rather than kept as a separate mechanism.

Known limitations

  • ENVTESTPATH's arch selection has a separate, pre-existing bug — tracked as ENVTESTPATH arch-selection is decided at Makefile parse time, picks amd64 on any cold bin/ regardless of host arch #2377, not fixed here. Its ifeq-based amd64 fallback is decided at Makefile-parse time and picks the amd64-forced variant on any cold bin/, regardless of actual host arch — confirmed via direct repro (see ENVTESTPATH arch-selection is decided at Makefile parse time, picks amd64 on any cold bin/ regardless of host arch #2377). It's invisible on amd64 CI (no CI/release-branch risk) and only affects arm64 local dev with a cold checkout, so it's out of scope here rather than a reason to expand this PR's diff a third time.
  • golangci-lint@v1.54.2 (this branch's pin) doesn't build against a modern Go 1.26 toolchain — a pre-existing, unrelated x/tools incompatibility; CI pins its own Go version and isn't affected. Bumping to the latest v1.x (v1.64.8, avoiding v2's config/default-linter overhaul) does fix the build, but its updated staticcheck/ineffassign/gosimple linters immediately surface ~20 pre-existing findings across unrelated files. Left the pin as-is — real scope growth this PR shouldn't absorb.
  • Unversioned tool symlinks (bin/<tool> → last-branch-wins, pre-existing behavior), a lower-priority observation, also worth a follow-up.

Testing

Verified against GNU Make 4.4.1 specifically (installed via Homebrew), not just the macOS-default Make 3.81 — the latter silently ignores .SHELLFLAGS, so it can't exercise set -e recipe behavior at all.

  • rm -rf bin/oadp-1.4 / rm -f bin/setup-envtest, then gmake controller-gen / gmake kustomize / gmake envtest individually — each installs fresh.
  • Re-running each immediately after → "already installed", no reinstall, no network call.
  • Simulated a wrong-arch binary (cross-compiled linux/amd64) for controller-gen, kustomize, and envtest → each correctly detected via go version -m's GOARCH/GOOS fields, removed, reinstalled with a valid native binary.
  • gmake generate manifests bundle → zero diff against a clean checkout.
  • gmake test → exit 0, all packages pass, Go code is formatted, api is up to date, bundle is up to date.

Note

Responses generated with Claude

Cherry-pick of the fix from oadp-dev. go-install-tool-branch only
installs when the binary is missing, never verifying the pinned
version against what's already on disk. Once a binary lands in
bin/<branch>/, it's reused forever across branch switches and version
bumps since bin/ is gitignored and nothing else resets it. kustomize
and controller-gen's targets also weren't fully .PHONY (only the
wrapper name was, not the binary path), so Make's own mtime-based
staleness check could skip their recipe entirely before any version
check ran.

Introduce go-install-tool-versioned, which compares a sidecar
<binary>.version marker file against the pinned version instead of
introspecting the binary's own --version output. Binary introspection
isn't reliable for every tool installed this way: kustomize's `version`
command depends on ldflags its own release process sets, which `go
install` doesn't set, so identically-installed kustomize binaries were
observed reporting "(devel)", an unexpanded `$Format:%H$` git-archive
placeholder, or a correct version string depending on unrelated
build-time factors.

oadp-1.4's golangci-lint target didn't have any version-check at all
(unlike oadp-dev's, which had a partial/broken one), so it's folded
into the same fix here.

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
@coderabbitai

coderabbitai Bot commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: cf286123-39fe-420a-a0d7-d3f55f763a84

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Comment @coderabbitai help to get the list of available commands.

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 10, 2026
Joeavaikath
Joeavaikath previously approved these changes Aug 10, 2026
@codecov-commenter

codecov-commenter commented Aug 10, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 81.25000% with 15 lines in your changes missing coverage. Please review.
✅ Project coverage is 38.47%. Comparing base (42938ce) to head (b4eeaa4).
⚠️ Report is 4 commits behind head on oadp-1.4.

⚠️ Current head b4eeaa4 differs from pull request most recent head d3ca37d

Please upload reports for the commit d3ca37d to get more accurate results.

Files with missing lines Patch % Lines
api/v1alpha1/zz_generated.deepcopy.go 0.00% 12 Missing ⚠️
pkg/common/common.go 95.16% 2 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##           oadp-1.4    #2368      +/-   ##
============================================
+ Coverage     37.79%   38.47%   +0.67%     
============================================
  Files            30       30              
  Lines          5033     5113      +80     
============================================
+ Hits           1902     1967      +65     
- Misses         2935     2949      +14     
- Partials        196      197       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

This branch's check-envtest-arch (landed via openshift#2100, Feb 2026) has been
unconditionally reinstalling setup-envtest on every single
`make test`/`make envtest` invocation since it merged: it treats any
`$(ENVTEST) --help` failure as "wrong architecture," but setup-envtest's
own --help exits 2 by its own convention even on a perfectly working
binary. Since check-envtest-arch is itself .PHONY and a prerequisite of
$(ENVTEST), that always-nonzero exit forces a delete+reinstall on every
invocation, unconditionally, defeating the point of caching this binary
at all.

Verified by cross-compiling a real linux/amd64 setup-envtest and running
it on this darwin/arm64 host: the shell reports exit code 126
specifically (POSIX "found but cannot execute" / exec format error) for
a genuinely incompatible binary, distinct from the tool's own exit
codes. Check that instead.

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
@kaovilai kaovilai changed the title [oadp-1.4] Make controller-gen/kustomize/golangci-lint version checks reliable [oadp-1.4] Make controller-gen/kustomize/golangci-lint/envtest tool-binary caching reliable Aug 12, 2026
@openshift-ci

openshift-ci Bot commented Aug 12, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kaovilai

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

1 similar comment
@openshift-ci

openshift-ci Bot commented Aug 12, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kaovilai

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

kaovilai and others added 2 commits August 12, 2026 09:55
Cherry-pick of the same extension from oadp-dev. The version-marker
check alone isn't enough: a binary can have a correct .version marker
but still be the wrong architecture, e.g. if a containerized build
with a different GOARCH bind-mounts the host's bin/ directory (this
Makefile documents exactly that workflow for `make test`: `docker run
--platform linux/amd64 -v $PWD:$PWD ...`). Anything that does
`go install` in there writes onto the host's real bin/ tree since
it's the same mounted path, not a copy. Not envtest-specific — it can
happen to any of these four cached tool binaries.

go-install-tool-versioned now also probes `$(1) --version` and checks
specifically for exit code 126 (POSIX "found but cannot execute" /
exec format error), same technique as the envtest fix already on this
branch. Verified controller-gen and kustomize (this branch's v4.5.5
pin) correctly detect and repair a wrong-arch binary even when its
.version marker already matches the pinned version: cross-compiled a
real linux/amd64 binary, copied it over the working native binary,
confirmed each was detected and replaced. golangci-lint shares the
identical code path but couldn't be exercised directly on this branch
(its v1.54.2 pin fails to build against this sandbox's Go 1.26
toolchain, a pre-existing environment issue unrelated to this change).

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
Cherry-pick of the same fix from oadp-dev. An independent second-opinion
review caught a severe bug in the previous commit: the exit-126 probe
(`$(1) --version`, then a separate `if [ $? -eq 126 ]`) is a bare
command outside any &&/||/if guard. Under `set -e` — which this
Makefile's `.SHELLFLAGS = -ec` enables, honored by GNU Make 3.82+ — a
bare failing command aborts the whole recipe immediately, before the
exit-code check ever runs. My local macOS Make (3.81) silently ignores
.SHELLFLAGS, so none of my testing could have caught this.

Confirmed for real: installed GNU Make 4.4.1 via Homebrew and
reproduced the crash directly on this branch too — `make kustomize`
died with "Error 1" every time the binary already existed, because
kustomize's own `--version` exits 1 by its own convention even on a
perfectly healthy binary (this branch's kustomize v4.5.5 pin, same
issue as v5.2.1 on oadp-dev).

Replaced the whole exit-code-probe approach with `go version -m`,
which reads a binary's embedded module version and GOOS/GOARCH
directly from its build info, without executing it at all:
  - No execution means no exit-code heuristic to get wrong, and no
    -e hazard, for any of these four tools.
  - No execution also closes a gap the review surfaced: exit-126
    detection cannot work at all inside a container with
    qemu-user-static/binfmt_misc registered (standard in multi-arch
    CI/build images), since a wrong-arch binary just runs under
    emulation and returns its own exit code instead of an
    exec-format-error.
  - Drops the sidecar `.version` marker file entirely — go version -m
    reads the real, authoritative module version already embedded in
    the binary.

This branch's check-envtest-arch (a separate target here, unlike
oadp-dev's inline check) is replaced outright by a plain call to the
same go-install-tool-versioned macro used by the other three tools.

Verified end-to-end with GNU Make 4.4.1: fresh install, idempotent
re-run (previously the exact crash case for kustomize/envtest), and
wrong-arch detection+repair, for controller-gen/kustomize/envtest.
golangci-lint@v1.54.2 still can't be exercised directly on this branch
(pre-existing build incompatibility with this sandbox's Go 1.26
toolchain, unrelated to this change — confirmed by testing whether a
newer v1.x pin would help: it does fix the build, but its updated
default linter set immediately surfaces ~20 pre-existing findings
across unrelated files, which is real scope growth this PR shouldn't
absorb; leaving the pin as-is). `make generate manifests bundle` zero
diff, `make test` fully green.

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
ifeq evaluates its $(shell ...) condition at parse time, before any
target's prerequisites run. On a cold bin/ (setup-envtest not yet
installed), `$(ENVTEST) list` fails silently, the grep finds nothing,
and ENVTESTPATH gets permanently redefined to force --arch=amd64 --
regardless of host arch -- even though setup-envtest is correctly
installed for the host's native arch by the time the `test` recipe
actually runs.

Move the fallback entirely into the shell expression itself (still a
recursively-expanded variable, so it's only evaluated when referenced
in the `test` recipe, after `envtest` has installed the right arch).

Fixes openshift#2377

Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
…in/"

This reverts commit 999e0fb.

Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
Both go-install-tool-versioned's doc comment and the envtest section's
note narrated the debugging process that led here (earlier attempts,
"confirmed with real GNU Make 4.4.1", "the previous exit-code-126
version... reliably crashed", a specific past PR reference) rather
than just stating why the current code is shaped this way. Keeps the
durable WHY (unreliable tool self-report, the set -e hazard as a
standing fact about this Makefile, the qemu/binfmt_misc blind spot)
as present-tense design rationale instead.

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
@openshift-ci

openshift-ci Bot commented Aug 12, 2026

Copy link
Copy Markdown

@kaovilai: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/unit-test d3ca37d link true /test unit-test

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants