Make the macOS compatibility job produce usable signal again - #194
Merged
Conversation
The two macOS checks were set up so that if one failed, the other was cancelled straight away. GitHub then shows the cancelled one as failed, so a check that was passing when it got stopped looks broken. That sent me down the wrong path twice today, both times concluding the older macOS was broken when it had actually been cancelled partway through while passing. You could only tell by opening the raw log. Both now run to the end and report for themselves.
Seventeen tests time out on the older macOS runner, the same seventeen every time. They are not broken on that version of macOS. Giving them four times longer to finish changed nothing, and in the same run the machine also fails to open plain network sockets and to read output from programs it started. Those have nothing in common with each other except needing the machine to get around to them. So the runner stalls partway through and starves whatever is waiting. The suite always runs in the same order, which is why it is always the same seventeen. They are skipped on that runner only. All of them still run on the newer macOS and on every pull request, so nothing stops being checked. The point is that the older runner can now come back green, and go red only when something is actually wrong.
arzafran
changed the title
This was referenced Jul 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
The macOS compatibility job has been red on every run since at least 21 July, across every branch. Two separate problems were keeping it that way, and neither was a real bug in the app.
1. The two macOS legs cancelled each other.
fail-fast: truemeant a failure on one killed the other, and GitHub reports a cancelled leg as a failure togh pr checks. A leg that was passing when it got stopped read as broken. That produced two wrong conclusions in one afternoon, both only detectable by opening the raw log and finding##[error]The operation was canceledafter a run of passing tests.2. Seventeen tests time out on the macos-15 runner — the same seventeen every run, verified by intersecting failure sets across independent runs. They are not broken on macOS 15. They are skipped on that leg only, and still run everywhere else.
Together: the job can go green again, and go red only when something is actually wrong.
Why these seventeen are not a macOS 15 bug
@available, no#available, noNSAppKitVersionchecks. A genuinely unguarded macOS-26-only symbol could not compile here at all, since Swift checks availability againstMACOSX_DEPLOYMENT_TARGET(14.0).ciScale— 3s became 12s, 10s became 40s — and the same tests timed out. A 12-second wait expiring is not a slow overlay.TerminalControllerSocketSecurityTeststimes out waiting for a raw Unix domain socket to appear; subprocess-log waits time out too. A BSD socket accept, aDispatchQueue.main.asyncclosure, and anXCTNSPredicateExpectationpoll share no SwiftUI or AppKit code. The only thing in common is needing an async completion serviced promptly.No coverage is lost
Every quarantined test still runs on the macos-26 compat leg and in the main CI workflow's
unit-testsjob on every PR. The quarantine is scoped to one leg of one workflow. That includes the fiveTerminalControllerSocketSecurityTestsentries, which are security tests and would otherwise be the most alarming thing on the list.Summary
fail-fast: falseon the compat matrix, with a comment recording why so it is not "optimised" back.QUARANTINED_ON_COMPATinscripts/ci-run-unit-tests.shlists the 17 with the full evidence inline; applied only whenPROGRAMA_UNIT_TEST_QUARANTINEis set.quarantine: trueon the macos-15 leg only, alongside the existingsmoke/skip_zigflags.This is a workaround for CI infrastructure, not a fix. The code comment says so and names the conditions to revisit under.
Test plan
Verified locally by stubbing
PROGRAMA_XCODEBUILD_COMMAND=echoand inspecting the generated argument list:-skip-testing:flags (1 pre-existing + 17 quarantined), andQuarantining 17 runner-starvation-prone testsis logged-skip-testing:flag, no quarantine notice — confirming it cannot leak into other jobsbash -n scripts/ci-run-unit-tests.shpassesworkflow-guard-testspassescancelled