tests: speed up unit tests by running them concurrently#5980
Draft
Bobronium wants to merge 6 commits into
Draft
Conversation
Co-authored-by: Claude <noreply@anthropic.com>
Bobronium
force-pushed
the
arseny/concurrent-tests
branch
from
4cb205f to
d27a648
Compare
…ture The shared event loop couples timing across a concurrent group, so the tight speed_factor=5.0 pacing in test_user_turn_exceeded flaked. Add an is_concurrent fixture (a single isinstance on the test node) and use speed = 3.0 if is_concurrent else 5.0, relaxing pacing only on the concurrent path and leaving sequential runs unchanged. Co-authored-by: Claude <noreply@anthropic.com>
The text-input turn is scheduled out-of-band, so on a shared event loop run_session's teardown can race its reply. Bump drain_delay to 2.0 only when is_concurrent, leaving the sequential path at 0.5. Co-authored-by: Claude <noreply@anthropic.com>
Its text-input interruption is coupled to the play-out clock (the interrupted message only commits once the synchronized transcript is non-empty), so it can't run on the shared concurrent loop. Mark it no_concurrent and widen drain_delay so the out-of-band reply also clears the teardown margin when run sequentially. Reverts the earlier is_concurrent attempt. Co-authored-by: Claude <noreply@anthropic.com>
Bring the deterministic virtual-time harness onto the concurrent-tests branch and mark every time-dependent module no_concurrent. - add tests/virtual_time.py + wire event_loop_policy / _virtual_wall_clock / marker registration into conftest; add async-solipsism dev dep - virtual_time modules run no_concurrent: a concurrent group shares one event loop, which bypasses the per-test autojump loop and races the process-global clock patch + validator swap (test_agent_session fails outright concurrent; others just lose the speedup). Measured concurrent vs --no-concurrent per module: no_concurrent is faster or equal everywhere, dramatically so for the heavy suites. - revert the concurrency-driven relaxations now that these run sequentially: restore test_user_turn_exceeded (speed=5.0, drop is_concurrent param) and test_agent_session (drain_delay, drop single-test no_concurrent) to strict bodies; remove the now-unused is_concurrent fixture - mark module-level no_concurrent on all 17 virtual_time modules - suppress the event_loop_policy override deprecation warning Co-authored-by: Claude <noreply@anthropic.com>
- remove the unit->concurrent auto-promotion (pytest_collectstart); every async unit module now declares concurrent/no_concurrent explicitly. Mark the 15 async unit modules concurrent (incl. test_interruption/test_interruption_ failover); leave sync-only modules as plain unit (concurrency is async-only, so a concurrent marker there is a no-op). - fix the virtual clock leaking into non-loop callers: _now/_perf returned a frozen 1.7e9 epoch when no loop was running, so any background thread (e.g. the OpenTelemetry metrics exporter) reading time during a virtual-time test got a 2023 timestamp the gateway rejects. Scope virtual time to the async-solipsism loop and return the real clock everywhere else. Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Running
pytest --unitwith these changes goes from 3 minutes 44 seconds to 44 seconds.Just
test_agent_session.pyfinishes under 8 seconds instead of 1 minute 45 seconds.How does it work
New plugin will group all tests marked as concurrent by their module and execute them in a shared event loop. This shrinks wall time to the slowest test + raw compute. This also preserves some level of isolation.
contextvarsare used to preserve input capture and track leaked tasks.All unit tests are marked as
concurrentby default. Tests from other categories are unaffected, unless explicit--concurrentoption is used when launching pytest.To disable concurrent execution,
--no-concurrentoption can be used.To disable per-module or per-test,
pytest.mark.no_concurrentcan be used.Changes
pytest-asyncio-concurrent— gives the speeduptests/concurrent.pythat extends onpytest-asyncio-concurrentand adds crucial features:pytest)fail_on_leaked_taskscontinues to work)--concurrent/--no-concurrentoptionstests/test_ipc.pyandtests/test_audio_decoder.pyas concurrentNo tests were changed.
This PR:
pytest --unit 742 passed, 25 deselected, 632 warnings in 44.33smain:pytest --unit 742 passed, 25 deselected, 645 warnings in 224.53s (0:03:44)pytest --unit -n 4 742 passed, 648 warnings in 116.79s (0:01:56)