Skip to content

Disable the Hypothesis deadline on the copartitioned-assignor property tests - #752

Open
wbarnha wants to merge 2 commits into
masterfrom
claude/fix-pypy-hypothesis-deadline
Open

Disable the Hypothesis deadline on the copartitioned-assignor property tests#752
wbarnha wants to merge 2 commits into
masterfrom
claude/fix-pypy-hypothesis-deadline

Conversation

@wbarnha

@wbarnha wbarnha commented Aug 4, 2026

Copy link
Copy Markdown
Member

Description

The Python pypy3.11/Cython: false CI leg fails intermittently on tests/meticulous/assignor/test_copartitioned_assignor.py, with an error that is about timing rather than about assignment correctness:

hypothesis.errors.FlakyFailure: Hypothesis test_remove_clients(
    partitions=243, replicas=62, num_clients=686, num_removal_clients=1
) produces unreliable results: Failed on the first call but did not on a
subsequent one

Unreliable test timings! On an initial run, this test took 5061.03ms, which
exceeded the deadline of 4000.00ms, but on a subsequent run it took
3483.45ms, which did not. If you expect this sort of variability in your test
timings, consider turning deadlines off for this test by setting
deadline=None.

These four property tests assert that the assignment produced is valid, not that it is produced within any particular time. PyPy's JIT warmup makes a single example's timing vary by well over a second, so a fixed 4000 ms deadline is tripped intermittently even though the assignment is correct. This sets TEST_DEADLINE = None; the job's own timeout still guards against a real hang.

Why this is worth landing on its own

This branch already existed with the fix but no PR, so the flake keeps failing unrelated PRs. It is not a hypothetical:

Both of those PRs currently carry 2f04895, a cherry-pick of this branch's commit, purely to get their CI green. Once this lands they can drop it, or it merges as a no-op since the change is identical.

Testing

Full suite passes (2207 passed, 4 skipped). The four assignor property tests pass with the deadline removed, and the PyPy leg is green on both PRs carrying the cherry-pick.

Branch was 15 commits behind; master has been merged in so CI runs against the current base.

🤖 Generated with Claude Code

https://claude.ai/code/session_019oX4oGCQGgvPJHabjwfaBk


Generated by Claude Code

wbarnha and others added 2 commits July 21, 2026 15:25
The copartitioned-assignor property tests run on PyPy since the PyPy
skips were removed. They carried a fixed 4000ms Hypothesis deadline, but
PyPy's JIT warmup makes a single example's timing vary by over a second,
so runs intermittently exceed it and Hypothesis reports DeadlineExceeded
/ FlakyFailure -- failing the (non-required) PyPy leg and turning master
CI red even though the assignment produced is valid.

These tests assert assignment correctness, not performance, so set
deadline=None (as the DeadlineExceeded message itself recommends). The
property assertions are unchanged and the job timeout still guards
against a real hang.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HHPL4VFWQRQPpjR1gXSKyL
@codecov

codecov Bot commented Aug 4, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.06%. Comparing base (803c7a4) to head (a9ec7d9).

Additional details and impacted files
@@           Coverage Diff           @@
##           master     #752   +/-   ##
=======================================
  Coverage   96.06%   96.06%           
=======================================
  Files         103      103           
  Lines       11072    11072           
  Branches     1191     1191           
=======================================
  Hits        10636    10636           
  Misses        345      345           
  Partials       91       91           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants