Skip to content

[WIP] CI flaky tests rerun and report#1665

Open
aleksandr-voitenko wants to merge 15 commits intostagingfrom
ci-flaky-test-rerun-and-report
Open

[WIP] CI flaky tests rerun and report#1665
aleksandr-voitenko wants to merge 15 commits intostagingfrom
ci-flaky-test-rerun-and-report

Conversation

@aleksandr-voitenko
Copy link
Copy Markdown
Collaborator

@aleksandr-voitenko aleksandr-voitenko commented Apr 8, 2026

Description

This PR improves retry behavior in OSN integration tests and adds CI reporting for flaky passes.

image

Motivation and Context

A pooled user account can become unusable after throttling or other third-party-side issues. When that happens, simply rerunning the same test with the same OBS state and the same credentials often repeats the failure. We need retry preparation, not just retry count.

This branch adds two pieces around that:

  • retry-aware test recovery for streaming-style suites
  • a dedicated CI check that reports tests which only passed after retrying

Main Changes

Retry preparation before the next attempt

Streaming-related suites now call retry preparation from afterEach, before Mocha starts the next attempt.

The recovery flow in obs_handler.ts

  • captures retry failure context
  • clears queued OBS output signals and stops active outputs
  • restores the known-good stream key for ordinary flaky cases
  • rotates to a fresh pooled user only for failures that look like bad remote session / credential state such as ConnectFailed or Disconnected

This is the key behavioral change: retries now start from a cleaner OBS state, and when the failure looks account-related, they can acquire fresh credentials instead of reusing a stale session.

User-pool

user_pool_handler.ts now supports marking the current reserved user as unhealthy and avoiding that user later in the same process.

That helps prevent the retry path from reacquiring the exact same account after it was already identified as non-working.

Retry-aware flaky reporting

list-reporter.js now:

  • logs retry attempts explicitly
  • records tests that eventually pass after retrying
  • publishes bounded flaky metadata through GitHub Actions step outputs

Dedicated GitHub check for flaky passes

The workflow now publishes a custom check after the test step in main.yml, using ci/publish-flaky-check.js.

Behavior:

  • runs even if the main test step failed
  • reports success when all tests passed first try
  • reports neutral when some tests only passed after retry

Negative test cleanup

The invalid-stream-key tests were wrapped so they always restore the original key in finally, which prevents them from contaminating later retries or later tests.

How Has This Been Tested?

CI, Windows and Mac

Types of changes

  • Tweak (non-breaking change to improve existing functionality)
  • Performance enhancement (non-breaking change which improves efficiency)
  • Code cleanup (non-breaking change which makes code smaller or more readable)

@aleksandr-voitenko aleksandr-voitenko changed the title [WIP] Ci flaky test rerun and report [WIP] CI flaky test rerun and report Apr 8, 2026
@aleksandr-voitenko aleksandr-voitenko changed the title [WIP] CI flaky test rerun and report [WIP] CI flaky tests rerun and report Apr 8, 2026
@aleksandr-voitenko aleksandr-voitenko force-pushed the ci-flaky-test-rerun-and-report branch from 5cd1be4 to 848a864 Compare April 8, 2026 18:27
@aleksandr-voitenko aleksandr-voitenko force-pushed the ci-flaky-test-rerun-and-report branch from 848a864 to e4e97a7 Compare April 9, 2026 16:25
@aleksandr-voitenko aleksandr-voitenko force-pushed the ci-flaky-test-rerun-and-report branch 2 times, most recently from 4133d1d to 6437af4 Compare April 9, 2026 18:36
@aleksandr-voitenko aleksandr-voitenko force-pushed the ci-flaky-test-rerun-and-report branch from 6437af4 to c648371 Compare April 9, 2026 18:54
@aleksandr-voitenko aleksandr-voitenko force-pushed the ci-flaky-test-rerun-and-report branch from 8b8d66b to 20e8e7e Compare April 10, 2026 14:24
@aleksandr-voitenko aleksandr-voitenko force-pushed the ci-flaky-test-rerun-and-report branch from 5b1d956 to 0551717 Compare April 10, 2026 15:27
@aleksandr-voitenko aleksandr-voitenko force-pushed the ci-flaky-test-rerun-and-report branch from 859efdc to 13429c5 Compare April 10, 2026 16:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants