Make the compatibility check actually check compatibility - #199
Merged
Conversation
This job exists to answer one question: does Programa build and launch on a macOS version other than the one we ship from. It has never answered it. The full test suite ran first and had to pass before the build and launch steps were allowed to run. Those tests fail on the older machine for reasons that have nothing to do with the operating system, so the build and launch steps were skipped every single time, for over a week. Five separate attempts to quiet the failing tests each fixed what was in front of them and uncovered the next thing. Building and launching now runs first and decides whether the job passes. The test suite still runs afterwards and still reports, but no longer decides, because it is the same suite that already runs on every pull request and re-running it on another macOS version was never the point.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
This job exists to answer one question: does Programa build and launch on a macOS version other than the one
release.ymlships from? It has never answered it.The full ~1,500-test unit suite ran first and had to pass before the build and smoke steps were allowed to run. Those tests fail on the macos-15 runner for reasons unrelated to the OS, so the steps that matter were skipped on every single run:
That has been the state since at least 21 July. Five successive attempts to quiet the failing tests (#193 timeout scaling, #194 17 methods, #197 7 classes, #198 8 classes) each fixed what was in front of them and revealed the next thing — while the actual compatibility signal stayed unreachable behind them.
The change
Build and smoke now run first and gate the job. Unit tests run afterwards,
continue-on-error, and report without deciding.Two reasons that ordering is right, not just pragmatic:
ci.yml. Re-running it on two more macOS versions produced no unique signal.Also fixes a coverage gap I found while doing this
matrix.smokeistrueonly on macos-15. macos-26's entire contribution was running unit tests — it never built or launched the app as a gating step. The new "Build app" step runs on both legs, so each now proves something only it can: macos-15 builds ghostty from Zig and launches the app under a virtual display; macos-26 builds against the macOS 26 SDK with the prebuilt xcframework.I did not enable
smokeon macos-26. It is disabled there for a reason I could not establish (that leg also setsskip_zig: truebecause "zig 0.15.2 MachO linker can't resolve libSystem on macOS 26"), and turning on a virtual-display smoke test I cannot verify would be guessing.Details worth reviewing
if: failure()→if: always()on the crash-report and xcresult-upload steps. With the unit-test stepcontinue-on-error, the job is green even when tests fail, sofailure()would never fire and the artifacts would silently stop being collected.::warning::annotation, so a failing unit suite is still visible in the run summary rather than buried in a green job.What this does not do
It does not fix the underlying starvation, the
WorkspaceStressProfileTestshost crash, or macos-26's intermittentWindowBrowserSlotViewTestsfailure. Those remain open and are now visible as warnings instead of a permanently red check nobody reads.Test plan
tests/test_ci_unit_test_runner_behavior.shpasses (the existing guard on this runner script)- name:, 8 for keys) — no YAML validator available locally, so checked structurallyworkflow-guard-testspasses