CI: De-flake coverage upload and stop fail-fast cancelling the matrix#845
Open
mmcky wants to merge 3 commits into
Open
CI: De-flake coverage upload and stop fail-fast cancelling the matrix#845mmcky wants to merge 3 commits into
mmcky wants to merge 3 commits into
Conversation
Two unrelated infra issues have been turning CI red on green code: 1. The Coveralls step runs on all three Linux matrix jobs and uploads to the same build with no parallel coordination. Whichever Linux job reports after the build is closed gets "422 Build is already closed" and fails. Restrict the upload to a single Linux job (3.13) and mark it continue-on-error so a Coveralls outage never blocks a merge. 2. The matrix used the default fail-fast: true, so any single job failure (the 422 above, or a flaky wall-clock timing test) cancelled every other in-progress job and painted the whole matrix red. Set fail-fast: false so jobs run to completion independently. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR hardens the GitHub Actions CI workflow by preventing flaky/non-test infrastructure failures (Coveralls upload race, and default matrix fail-fast behavior) from incorrectly turning the overall CI status red.
Changes:
- Set the test job’s matrix strategy to
fail-fast: falseso one failure doesn’t cancel the rest of the matrix. - Restrict Coveralls uploads to a single Linux/Python job and mark the upload step
continue-on-error: trueto avoid intermittent 422/upload outages failing CI.
- Quote the matrix python-version entries as strings ("3.12"/"3.13"/
"3.14"). Bare YAML numbers are brittle (3.10 would parse as 3.1) and
make the ``matrix.python-version == '3.13'`` Coveralls gate unreliable.
- Pin coverallsapp/github-action to @v2 instead of @master, matching the
major-tag pinning used by the other actions in this file, for
reproducibility and supply-chain safety.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
Author
|
@oyamad this should fix the flaky coveralls issue we are having. I'll merge this once green but let me know if you have any issues. |
oyamad
reviewed
Jun 28, 2026
Comment on lines
59
to
67
| - name: Coveralls | ||
| uses: coverallsapp/github-action@master | ||
| if: runner.os == 'Linux' | ||
| uses: coverallsapp/github-action@v2 | ||
| # upload from one Linux job only (avoids the Coveralls parallel-build 422 | ||
| # race); continue-on-error so an upload hiccup can't fail CI | ||
| if: runner.os == 'Linux' && matrix.python-version == '3.13' | ||
| continue-on-error: true | ||
| with: | ||
| github-token: ${{ secrets.GITHUB_TOKEN }} | ||
| path-to-lcov: coverage.lcov |
Member
There was a problem hiding this comment.
ChatGPT says path-to-lcov is deprecated.
It suggested the following:
- name: Coveralls
uses: coverallsapp/github-action@v2
if: runner.os == 'Linux' && matrix.python-version == '3.13'
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
file: coverage.lcov
format: lcov
fail-on-error: false
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
CI has been going red on green code. Investigating the failure on #842 (and the matching failures on recent
mainpushes) turned up two independent, pre-existing infra problems — neither is a real test or code failure. This PR hardens the workflow against both.Problem 1 — Coveralls 422 race
The
Coverallsstep runs on all three Linux matrix jobs (if: runner.os == 'Linux') and each uploads to the same Coveralls build with no parallel coordination. Whichever Linux job reports after the build has been closed gets:On the #842 run,
ubuntu-3.14finished first and closed the build;ubuntu-3.13reported a minute later → 422 → job failed. Its pytest step had already passed. This is timing-dependent and intermittent — earlier runs on the same branch were green.Problem 2 —
fail-fastcascadeThe matrix had no
fail-fastkey, so it defaulted totrue. The moment one job failed (the 422 above, or the flaky timing test below), GitHub cancelled every other in-progress job — so a single flake painted the whole 9-job matrix red, with most jobs showingcancelledrather than their real result.Fix
matrix.python-version == '3.13')continue-on-error: trueon the Coveralls stepfail-fast: falseon the matrixCoverage is unaffected in practice — this is a pure-Python library, so line coverage does not vary across OS/Python, and one uploader is sufficient.
Note: a separate flaky test (not fixed here)
While diagnosing this I found that recent
mainfailures were caused by a different flake — a wall-clock timing assertion,quantecon/util/tests/test_timing.py::TestTimer::test_timeit_lambda_function, which overshoots on a slow runner (ACTUAL 0.207vsDESIRED 0.05,rtol=2). Withfail-fast: falsethat flake will no longer cancel the rest of the matrix, but the test itself can still go red on an unlucky runner. Tightening or de-timing that test is left as a follow-up so this PR stays focused on the workflow.🤖 Generated with Claude Code