Skip to content

Skip whole test groups on the older runner, not single tests - #197

Merged
arzafran merged 1 commit into
mainfrom
ci/quarantine-by-class
Jul 29, 2026
Merged

Skip whole test groups on the older runner, not single tests#197
arzafran merged 1 commit into
mainfrom
ci/quarantine-by-class

Conversation

@arzafran

Copy link
Copy Markdown
Member

What this does

Switches the macos-15 quarantine from 19 individual test methods to the 7 test classes they belong to.

The first version (#194) listed methods, derived from intersecting the failure sets of two independent runs. The next run failed two more — testWorkspaceCloseRejectsPinnedWorkspace and testPortalDropZoneOverlayPersistsAcrossVisibilityChanges — both in classes already partly on the list.

That will keep happening. Which individual methods lose the scheduler lottery varies per run; which classes they belong to does not. Listing methods guarantees a slow leak of new stragglers and a new PR every few days. Listing classes ends it.

Result so far

#194 took macos-15 from 17 failures to 2. This should take it to 0.

Why these are not macOS 15 bugs

Unchanged from #194, and worth restating because quarantining security-relevant tests deserves the scrutiny:

  • No OS-version-conditional code anywhere in the paths they cover. A genuinely unguarded macOS-26-only symbol could not compile, since Swift checks availability against MACOSX_DEPLOYMENT_TARGET (14.0).
  • 4x more time changed nothing (Give the slow-runner test waits the same headroom as everything else #193 raised every wait via ciScale; 3s became 12s, 10s became 40s). A 12-second wait expiring is not a slow overlay.
  • Unrelated subsystems fail together — raw Unix-socket waits and subprocess-log waits fail in the same run. A BSD socket accept and a DispatchQueue.main.async closure share no SwiftUI or AppKit code; the only thing in common is needing an async completion serviced promptly.
  • The slower runner passes. macos-26 is ~2x slower than macos-15 per the matrix comment, and it is green. That rules out "needs more time".

No coverage is lost

Every class still runs on the macos-26 compat leg and in the main CI workflow's unit-tests job on every PR. The quarantine is scoped to one leg of one workflow. That includes TerminalControllerSocketSecurityTests, which is security-relevant and would otherwise be the most alarming entry.

The classes

Class Defined in
BrowserWindowPortalLifecycleTests BrowserPanelTests.swift
CLINotifyProcessIntegrationTests WorkspaceRemoteConnectionTests.swift
GhosttySurfaceOverlayTests TerminalAndGhosttyTests.swift
NotificationDockBadgeTests NotificationAndMenuBarTests.swift
TerminalControllerSocketSecurityTests TerminalControllerSocketSecurityTests.swift
TerminalNotificationDirectInteractionTests TerminalAndGhosttyTests.swift
TerminalWindowPortalLifecycleTests TerminalAndGhosttyTests.swift

All build real NSWindows and wait on async completions — the common shape.

Test plan

  • bash -n scripts/ci-run-unit-tests.sh passes
  • Flag set → 8 -skip-testing: flags (1 pre-existing + 7 classes), logs Quarantining 7 runner-starvation-prone test classes
  • Flag unset → 1 flag, no quarantine notice — confirming it cannot leak into ci.yml
  • All 7 class names resolve to real classes in programaTests/. A typo'd -skip-testing target is silently ignored by xcodebuild, so this was worth checking explicitly.
  • macos-15 compat goes green

The list of tests to skip on the older macOS runner was written out one
test at a time, taken from what failed in two runs. The next run failed
two more, both in groups already on the list.

That will keep happening. Which individual tests get starved changes run
to run; which groups they belong to does not. So the list is now seven
groups instead of nineteen tests, and it stops needing a new entry every
few days.

All of them still run on the newer macOS and on every pull request, so
nothing stops being checked.
@arzafran
arzafran merged commit 6b310e7 into main Jul 29, 2026
7 of 8 checks passed
@arzafran
arzafran deleted the ci/quarantine-by-class branch July 29, 2026 22:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant