[tests] Disable the AppSizeTest on branches using an Xcode beta.#26037
Conversation
On branches that use beta versions of Xcode, we typically do a lot of binding work, and that mean that frequent updates to the known failures for this test are necessary. So frequent that it becomes frustrating, and also slows down pace, because there will be a lot of merge conflicts between PRs that all update the known failures for this test. So just disable it when using a beta version of Xcode. Once the branch switches to a stable version of Xcode, there will be much less binding work, and we can resume this test.
There was a problem hiding this comment.
Pull request overview
This PR disables AppSizeTest when the test suite is running against a non-stable (beta/preview) Xcode, to avoid frequent churn and merge conflicts in the app size “known failures/expected” data during periods of heavy binding work.
Changes:
- Add an early
Assert.Ignoregate in the sharedRun(...)helper whenConfiguration.XcodeIsStableis false. - Ensure the ignore message clearly documents why the test is being skipped under beta Xcode.
✅ API diff for current PR / commitNET (empty diffs)✅ API diff vs stableNET (empty diffs)ℹ️ Generator diffGenerator Diff: vsdrops (html) vsdrops (raw diff) gist (raw diff) - Please review changes) Pipeline on Agent |
🚀 [CI Build #a16bb43] Test results 🚀Test results✅ All tests passed on VSTS: test results. 🎉 All 199 tests passed 🎉 Tests counts✅ assembly-processing: All 1 tests passed. Html Report (VSDrops) Download macOS tests✅ Tests on macOS Monterey (12): All 5 tests passed. Html Report (VSDrops) Download Linux Build VerificationPipeline on Agent |
On branches that use beta versions of Xcode, we typically do a lot of binding
work, and that mean that frequent updates to the known failures for this test
are necessary. So frequent that it becomes frustrating, and also slows down
pace, because there will be a lot of merge conflicts between PRs that all
update the known failures for this test.
So just disable it when using a beta version of Xcode. Once the branch
switches to a stable version of Xcode, there will be much less binding work,
and we can resume this test.