fix(ci): harden apt-get against mirror hangs in eBPF/vrf-module steps - #435
Closed
privateip wants to merge 1 commit into
Closed
fix(ci): harden apt-get against mirror hangs in eBPF/vrf-module steps#435privateip wants to merge 1 commit into
privateip wants to merge 1 commit into
Conversation
Bare "apt-get update"/"apt-get install" have no per-request timeout, so a stalled connection to GitHub's default regional mirror (azure.archive.ubuntu.com has been observed hanging outright) blocks until the job's own timeout-minutes kills it, burning the whole 15-25 minute budget instead of failing fast or retrying. This hit both #424 (install-ebpf-deps, inside the E2E Tests job) and #425 (the vrf kernel module step, inside Unit Tests (root)) in the same CI run window, on two otherwise-unrelated branches -- both jobs got cancelled at their timeout with apt still stuck on the initial 'update'. Add -o Acquire::Retries=3 -o Acquire::http::Timeout=10 -o Acquire::https::Timeout=10 to every apt-get invocation that installs CI-only packages (install-ebpf-deps' clang/llvm/ linux-libc-dev, and the two copies -- ci.yaml and scripts/ci.sh -- of the linux-modules-extra vrf-module install) so a bad mirror fails each attempt in ~10s and retries up to 3 times instead of hanging for the full job timeout. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Contributor
Author
|
Closing as redundant — this exact commit ( |
auto-merge was automatically disabled
August 20, 2026 14:16
Pull request was closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Both #424 and #425 hit the same CI failure in the same run window, on two otherwise-unrelated branches: an
apt-get update/apt-get installstep hung until the job'stimeout-minuteskilled it, rather than failing fast or retrying.install-ebpf-depscomposite action (clang/llvm/linux-libc-dev) hung ~25 minutes againstazure.archive.ubuntu.combefore falling back toarchive.ubuntu.com, then got cancelled at the job's 25-minute timeout withapt-get installnever completing.Unit Tests (root)job's "Load vrf kernel module" step (linux-modules-extra-$(uname -r)) hung onapt-get update -qqfor the full 15-minute timeout with no output at all (suppressed by-qq).Root cause: none of this repo's three
apt-getcall sites set any retry/timeout options, so a single unreachable/slow mirror blocks indefinitely instead of failing over quickly.Fix
Add
-o Acquire::Retries=3 -o Acquire::http::Timeout=10 -o Acquire::https::Timeout=10to every CIapt-get update/apt-get installinvocation:.github/actions/install-ebpf-deps/action.yaml(clang/llvm/linux-libc-dev — used by every job).github/workflows/ci.yaml'stest-unit-rootjob (vrf kernel module)scripts/ci.sh'se2etestcase (same vrf kernel module install, for local/CI parity)This bounds each mirror attempt to ~10s and retries up to 3 times, so a bad mirror fails in well under a minute instead of consuming the entire job timeout.
Test plan
python3 -c "import yaml; ...",bash -n scripts/ci.sh)yamlfmt -lintclean🤖 Generated with Claude Code