Skip to content

test: fix flaky tests#6848

Open
matthewelwell wants to merge 4 commits intomainfrom
test/fix-flaky-tests
Open

test: fix flaky tests#6848
matthewelwell wants to merge 4 commits intomainfrom
test/fix-flaky-tests

Conversation

@matthewelwell
Copy link
Contributor

@matthewelwell matthewelwell commented Mar 4, 2026

Changes

Fixes 2 flaky unit tests:

  1. test_get_feature_states_for_identity - this test failed when the multivariate feature values were returned in reverse order since the test assumed they were returned in the order they were created.
  2. test_identities_endpoint_returns_value_for_segment_if_rule_type_percentage_split_and_identity_in_segment - this test failed if the returned value from get_hashed_percentage_value was between 0.99 and 0.995. The maths when creating the Condition then meant that the percentage value was 99, and hence the user was deemed outside of the segment. Since the logic for actually determining segment membership is covered elsewhere, I just set the percentage to 100 to guarantee that the identity will be in the segment.

How did you test this code?

Ran the tests 100 times consecutively (using pytest.mark.parametrise) to verify that the tests consistently passed.

@matthewelwell matthewelwell requested a review from a team as a code owner March 4, 2026 20:51
@matthewelwell matthewelwell requested review from gagantrivedi and removed request for a team March 4, 2026 20:51
@vercel
Copy link

vercel bot commented Mar 4, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

3 Skipped Deployments
Project Deployment Actions Updated (UTC)
docs Ignored Ignored Preview Mar 5, 2026 9:16am
flagsmith-frontend-preview Ignored Ignored Preview Mar 5, 2026 9:16am
flagsmith-frontend-staging Ignored Ignored Preview Mar 5, 2026 9:16am

Request Review

@github-actions github-actions bot added api Issue related to the REST API testing labels Mar 4, 2026
@github-actions
Copy link
Contributor

github-actions bot commented Mar 4, 2026

Docker builds report

Image Build Status Security report
ghcr.io/flagsmith/flagsmith-e2e:pr-6848 Finished ✅ Skipped
ghcr.io/flagsmith/flagsmith-frontend:pr-6848 Finished ✅ Results
ghcr.io/flagsmith/flagsmith-private-cloud:pr-6848 Finished ✅ Results
ghcr.io/flagsmith/flagsmith-api-test:pr-6848 Finished ✅ Skipped
ghcr.io/flagsmith/flagsmith-api:pr-6848 Finished ✅ Results
ghcr.io/flagsmith/flagsmith:pr-6848 Finished ✅ Results

@github-actions
Copy link
Contributor

github-actions bot commented Mar 4, 2026

Playwright Test Results (oss - depot-ubuntu-latest-16)

passed  10 passed

Details

stats  10 tests across 7 suites
duration  8.9 seconds
commit  47c6c52
info  🔄 Run: #15129 (attempt 1)

@github-actions
Copy link
Contributor

github-actions bot commented Mar 4, 2026

Playwright Test Results (oss - depot-ubuntu-latest-arm-16)

passed  10 passed

Details

stats  10 tests across 7 suites
duration  12.7 seconds
commit  47c6c52
info  🔄 Run: #15129 (attempt 1)

@github-actions
Copy link
Contributor

github-actions bot commented Mar 4, 2026

Playwright Test Results (private-cloud - depot-ubuntu-latest-arm-16)

passed  16 passed

Details

stats  16 tests across 13 suites
duration  35.2 seconds
commit  47c6c52
info  🔄 Run: #15129 (attempt 1)

@github-actions
Copy link
Contributor

github-actions bot commented Mar 4, 2026

Playwright Test Results (private-cloud - depot-ubuntu-latest-16)

passed  1 passed

Details

stats  1 test across 1 suite
duration  52.5 seconds
commit  47c6c52
info  🔄 Run: #15129 (attempt 1)

@codecov
Copy link

codecov bot commented Mar 4, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.33%. Comparing base (43bb456) to head (47c6c52).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #6848   +/-   ##
=======================================
  Coverage   98.33%   98.33%           
=======================================
  Files        1336     1336           
  Lines       49615    49615           
=======================================
  Hits        48790    48790           
  Misses        825      825           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions github-actions bot added testing and removed testing labels Mar 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api Issue related to the REST API testing

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants