[WIP] CI flaky tests rerun and report#1665
Open
aleksandr-voitenko wants to merge 15 commits intostagingfrom
Open
[WIP] CI flaky tests rerun and report#1665aleksandr-voitenko wants to merge 15 commits intostagingfrom
aleksandr-voitenko wants to merge 15 commits intostagingfrom
Conversation
5cd1be4 to
848a864
Compare
848a864 to
e4e97a7
Compare
4133d1d to
6437af4
Compare
6437af4 to
c648371
Compare
8b8d66b to
20e8e7e
Compare
5b1d956 to
0551717
Compare
859efdc to
13429c5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR improves retry behavior in OSN integration tests and adds CI reporting for flaky passes.
Motivation and Context
A pooled user account can become unusable after throttling or other third-party-side issues. When that happens, simply rerunning the same test with the same OBS state and the same credentials often repeats the failure. We need retry preparation, not just retry count.
This branch adds two pieces around that:
Main Changes
Retry preparation before the next attempt
Streaming-related suites now call retry preparation from
afterEach, before Mocha starts the next attempt.The recovery flow in
obs_handler.tsConnectFailedorDisconnectedThis is the key behavioral change: retries now start from a cleaner OBS state, and when the failure looks account-related, they can acquire fresh credentials instead of reusing a stale session.
User-pool
user_pool_handler.tsnow supports marking the current reserved user as unhealthy and avoiding that user later in the same process.That helps prevent the retry path from reacquiring the exact same account after it was already identified as non-working.
Retry-aware flaky reporting
list-reporter.jsnow:Dedicated GitHub check for flaky passes
The workflow now publishes a custom check after the test step in
main.yml, usingci/publish-flaky-check.js.Behavior:
successwhen all tests passed first tryneutralwhen some tests only passed after retryNegative test cleanup
The invalid-stream-key tests were wrapped so they always restore the original key in
finally, which prevents them from contaminating later retries or later tests.How Has This Been Tested?
CI, Windows and Mac
Types of changes