Cap hypothesis work on PyPy so the PyPy CI leg stops timing out - #750
Open
wbarnha wants to merge 2 commits into
Open
Cap hypothesis work on PyPy so the PyPy CI leg stops timing out#750wbarnha wants to merge 2 commits into
wbarnha wants to merge 2 commits into
Conversation
The advisory PyPy leg has been running out of wall-clock. On PR #699 it stalled for 6m17s with no output after the last tests/integration/cli file and was killed at the 15-minute job cap; the next file to run is tests/meticulous/assignor/test_copartitioned_assignor.py, whose name never reached the log, meaning its first test never reported. That file is four hypothesis property tests at the default 100 examples each. All of them were skipped on PyPy until #716 removed the PyPy skips, so PyPy started running them for the first time. Hypothesis executes its own generation and shrinking in pure Python, which PyPy runs slowly, and the explicit deadline=4000 measures JIT warm-up rather than the assignor -- once an example trips it, hypothesis drops into a silent shrinking search that can burn minutes. For scale, the whole of tests/meticulous is 5.8s on CPython here, yet PyPy sat in it for over six minutes. The leg has no headroom to absorb that: on the same base commit master takes 11m51s of the 15-minute budget, and raising the cap (already bumped 10 -> 15 by #716) only defers the problem, so cap the work instead. Register a "pypy" hypothesis profile in tests/conftest.py -- 20 examples, no deadline, too_slow health check suppressed -- and load it only when running on PyPy. CPython legs keep the full-size profile, so property coverage is unchanged where it is cheap to get. An explicit deadline= on @settings overrides the active profile (verified), so the profile alone cannot clear the assignor deadline; TEST_DEADLINE is made None on PyPy for that reason. Verified: under a simulated PyPy interpreter the tests resolve to deadline=None and max_examples=20; tests/meticulous drops from 4.95s to 1.40s under the profile on CPython while the default run is unchanged; full suite 2217 passed; isort/black/flake8 clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017K8xAH8Z3xWKHhNRCG2mg1
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #750 +/- ##
==========================================
+ Coverage 95.97% 95.98% +0.01%
==========================================
Files 103 103
Lines 11072 11072
Branches 1191 1191
==========================================
+ Hits 10626 10628 +2
+ Misses 352 350 -2
Partials 94 94 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The advisory PyPy leg has been running out of wall-clock. On PR #699 it
stalled for 6m17s with no output after the last tests/integration/cli file
and was killed at the 15-minute job cap; the next file to run is
tests/meticulous/assignor/test_copartitioned_assignor.py, whose name never
reached the log, meaning its first test never reported.
That file is four hypothesis property tests at the default 100 examples
each. All of them were skipped on PyPy until #716 removed the PyPy skips,
so PyPy started running them for the first time. Hypothesis executes its
own generation and shrinking in pure Python, which PyPy runs slowly, and the
explicit deadline=4000 measures JIT warm-up rather than the assignor -- once
an example trips it, hypothesis drops into a silent shrinking search that
can burn minutes. For scale, the whole of tests/meticulous is 5.8s on
CPython here, yet PyPy sat in it for over six minutes.
The leg has no headroom to absorb that: on the same base commit master takes
11m51s of the 15-minute budget, and raising the cap (already bumped 10 -> 15
by #716) only defers the problem, so cap the work instead.
Register a "pypy" hypothesis profile in tests/conftest.py -- 20 examples, no
deadline, too_slow health check suppressed -- and load it only when running
on PyPy. CPython legs keep the full-size profile, so property coverage is
unchanged where it is cheap to get.
An explicit deadline= on @settings overrides the active profile (verified),
so the profile alone cannot clear the assignor deadline; TEST_DEADLINE is
made None on PyPy for that reason.
Verified: under a simulated PyPy interpreter the tests resolve to
deadline=None and max_examples=20; tests/meticulous drops from 4.95s to
1.40s under the profile on CPython while the default run is unchanged; full
suite 2217 passed; isort/black/flake8 clean.
Co-Authored-By: Claude Opus 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_017K8xAH8Z3xWKHhNRCG2mg1