Skip to content

(fix) tests: fix flaky TestTwistedConnection.test_connection_initialization#767

Open
mykaul wants to merge 1 commit intoscylladb:masterfrom
mykaul:fix/flaky-twisted-test-connection-init
Open

(fix) tests: fix flaky TestTwistedConnection.test_connection_initialization#767
mykaul wants to merge 1 commit intoscylladb:masterfrom
mykaul:fix/flaky-twisted-test-connection-init

Conversation

@mykaul
Copy link
Copy Markdown

@mykaul mykaul commented Mar 26, 2026

Motivation

TestTwistedConnection::test_connection_initialization is flaky in CI. It was observed failing on PyPy 3.11 + macOS x86 (Rosetta 2) in this CI run, but it can fail on any platform where the Twisted reactor's global running state leaks between tests.

The root cause: TwistedLoop.maybe_start() checks reactor.running to decide whether to spawn a new reactor thread. If a prior test leaves reactor.running == True, maybe_start() becomes a no-op, the reactor.run mock is never called, and the assertion at line 116 fails:

AssertionError: expected call not found.
Expected: run(installSignalHandlers=False)
  Actual: not called.

Change

Patch twisted.internet.reactor.running to False in TestTwistedConnection.setUp(), ensuring maybe_start() always enters the branch that spawns the reactor thread regardless of leaked global state. The patcher is properly stopped in tearDown().

Testing

  • All 10 tests in test_twistedreactor.py pass.
  • 1000 runs × 6 tests per run = 6000 individual test executions with zero failures (completed in ~16 seconds), confirming the flakiness is resolved.
Batch 1/10: 100 runs OK (600 tests total, 1.8s)
Batch 2/10: 100 runs OK (1200 tests total, 3.1s)
Batch 3/10: 100 runs OK (1800 tests total, 4.6s)
Batch 4/10: 100 runs OK (2400 tests total, 6.2s)
Batch 5/10: 100 runs OK (3000 tests total, 7.6s)
Batch 6/10: 100 runs OK (3600 tests total, 9.5s)
Batch 7/10: 100 runs OK (4200 tests total, 11.6s)
Batch 8/10: 100 runs OK (4800 tests total, 13.0s)
Batch 9/10: 100 runs OK (5400 tests total, 14.5s)
Batch 10/10: 100 runs OK (6000 tests total, 16.2s)
All 10 batches x 100 runs = 1000 runs (6000 individual tests) passed in 16.2s

@mykaul
Copy link
Copy Markdown
Author

mykaul commented Mar 26, 2026

reviewed with Opus 4.6 and GPT-5.4

@mykaul
Copy link
Copy Markdown
Author

mykaul commented Mar 26, 2026

Failure seen @ #748

Patch reactor.running to False in setUp() so that maybe_start() always
enters the branch that spawns the reactor thread. Without this, leaked
global reactor state from prior tests can leave reactor.running as True,
causing maybe_start() to skip thread creation and the reactor.run mock
to never be called — making the assertion in test_connection_initialization
fail intermittently.

Observed in CI on PyPy 3.11 + macOS x86 (Rosetta 2), where timing
differences make the reactor state leak more likely.
@mykaul mykaul force-pushed the fix/flaky-twisted-test-connection-init branch from 8db899e to 7aafd82 Compare March 26, 2026 14:50
@mykaul mykaul changed the title tests: fix flaky TestTwistedConnection.test_connection_initialization (fix) tests: fix flaky TestTwistedConnection.test_connection_initialization Mar 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant