Skip to content

chore(ci): cut PR wall-clock ~9.5min → ~5-6min — dedupe dogfood, shard it, cache lint's build - #3622

Merged
os-zhuang merged 2 commits into
mainfrom
claude/ci-performance-optimization-ms8ll6
Jul 27, 2026
Merged

chore(ci): cut PR wall-clock ~9.5min → ~5-6min — dedupe dogfood, shard it, cache lint's build#3622
os-zhuang merged 2 commits into
mainfrom
claude/ci-performance-optimization-ms8ll6

Conversation

@os-zhuang

@os-zhuang os-zhuang commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Why

Job-level timing of recent PR runs shows where the time actually goes:

Job Wall time Long pole
CI / Test Core 9m21s turbo run test --affected — 8m35s
CI / Dogfood Regression Gate 8m41s dogfood suite — 7m34s
Lint / TypeScript Type Check 6m09s turbo run build — 4m38s (no cache)
CodeQL ~4m analysis (parallel, acceptable)

Two structural problems dominate: the dogfood suite ran twice per core PR (once in its dedicated job, once again inside Test Core's affected set, in parallel on two runners), and lint's typecheck job is the only build-running job with no turbo cache, so it rebuilt the entire workspace from scratch on every run — the identical build takes 51s in ci.yml's cached Build Core job.

What

  1. Exclude @objectstack/dogfood from Test Core (both the PR --affected run and the push run). The dedicated Dogfood job — gated on the same core filter — remains the sole, unconditional runner of the suite. Verified locally: turbo unions inclusive filters and then subtracts ! negations, so --affected --filter=!@objectstack/dogfood is exactly "affected minus dogfood" (140 → 137 tasks, nothing else dropped).

  2. Shard the Dogfood job 2-way (vitest run --shard=N/2, ~60 independent test files, deterministic file-level partition). The shard arg passes through turbo and is part of the turbo task hash, so each shard caches independently (verified: distinct hashes). The objectstack verify CLI step is shard-independent and runs on shard 1 only. Turbo cache keys are shard-scoped so the two matrix jobs don't race on one cache entry.

  3. Add the missing turbo cache step to lint's typecheck job — same key scheme as ci.yml, so the fallback prefix also hits caches saved by main pushes.

  4. Add concurrency cancel-in-progress to lint.yml and codeql.yml (ci.yml already has it). Both trigger on every PR sync; superseded runs were burning runners and delaying the queue.

Expected effect

  • CI workflow: ~9.5min → ~5–6min (Test Core sheds the duplicated 7.5-min suite; Dogfood halves to ~4–5min per shard).
  • Lint workflow: ~6min → ~2–2.5min on warm cache.
  • Meaningful runner-minute savings per PR sync from de-duplication + cancellation.

Verification

  • turbo run test --affected --filter=!@objectstack/dogfood --dry=json with a spec change: dogfood excluded, all other affected tasks intact.
  • vitest run --shard=1/30 in packages/qa/dogfood: 3 files / 13 tests, green — sharding partitions and runs correctly.
  • Full-chain smoke of the exact CI command shape: turbo run test --filter=@objectstack/dogfood -- --shard=1/30 → 61 tasks successful, exit 0.
  • --dry=json task-hash comparison between --shard=1/2 and --shard=2/2: hashes differ, so per-shard turbo caching is safe.
  • All three YAML files parse.

⚠️ Action needed at merge time

If branch protection lists "Dogfood Regression Gate" as a required status check, it must be updated to the two sharded check names (Dogfood Regression Gate (1/2) / Dogfood Regression Gate (2/2)).

🤖 Generated with Claude Code

https://claude.ai/code/session_01ECTCrcCdZpCHw5zFSgcmGt

Generated by Claude Code

@claude
chore(ci): cut PR wall-clock ~9.5min → ~5-6min — dedupe dogfood, shar…
61585a3 
…d it, cache lint's build

Three data-driven fixes from job-level timing of recent PR runs:

1. Test Core re-ran the whole ~7.5-minute dogfood suite that the dedicated
   Dogfood job was already running in parallel (both gate on the same `core`
   filter). Exclude @objectstack/dogfood from both the PR --affected run and
   the push run; the Dogfood job remains the sole (and unconditional) runner
   of the suite. Verified locally: turbo unions inclusive filters and then
   subtracts `!` negations, so `--affected --filter=!@objectstack/dogfood`
   is exactly "affected minus dogfood" (140 → 137 tasks, nothing else lost).

2. The Dogfood job itself was the workflow's longest pole (7m34s for the
   ~60-file suite on one 4-vCPU runner). Shard it 2-way with vitest's
   deterministic file-level --shard, passed through turbo (pass-through args
   are part of the turbo task hash, so each shard caches independently —
   verified: distinct hashes per shard). The `objectstack verify` CLI step is
   shard-independent and runs on shard 1 only. NOTE: if branch protection
   requires "Dogfood Regression Gate", the required check must be renamed to
   the two sharded names.

3. lint.yml's "TypeScript Type Check" job runs a full workspace build but was
   the only build-running job with NO turbo cache step — it rebuilt everything
   from scratch every run (4m38s observed, vs 51s for the same build in
   ci.yml's cached Build Core). Add the same .turbo/cache step.

Also: lint.yml and codeql.yml had no concurrency group, so superseded runs of
both (they trigger on every PR sync) kept burning runners and delaying the
queue. Add the same cancel-in-progress policy ci.yml already documents.

Full-chain smoke: `turbo run test --filter=@objectstack/dogfood -- --shard=1/30`
runs green end-to-end (turbo → pnpm → vitest, 61 tasks OK).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ECTCrcCdZpCHw5zFSgcmGt
@vercel

vercel Bot commented Jul 27, 2026 •
edited
Loading

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
objectstack Ignored Ignored Jul 27, 2026 10:05am

Request Review

@github-actions github-actions Bot added documentation Improvements or additions to documentation tooling labels Jul 27, 2026
@os-zhuang
os-zhuang marked this pull request as ready for review July 27, 2026 10:31
@os-zhuang
os-zhuang merged commit 1676de0 into main Jul 27, 2026
16 checks passed
@os-zhuang
os-zhuang deleted the claude/ci-performance-optimization-ms8ll6 branch July 27, 2026 10:32
@os-zhuang os-zhuang mentioned this pull request Jul 27, 2026
Merged
os-zhuang added a commit that referenced this pull request Jul 27, 2026
@os-zhuang @claude
fix(ci): restore the stable "Dogfood Regression Gate" required-check …
e3eb93f 
…name (#3622 follow-up) (#3644)

INCIDENT: every open PR in the repo is BLOCKED — mergeable, all checks
green, merge button dead.

Root cause: #3622 sharded the dogfood job 2-way, so its checks now
publish as "Dogfood Regression Gate (1/2)" / "(2/2)". Branch protection
still requires the bare context "Dogfood Regression Gate", which no job
emits anymore — the requirement can never be satisfied. #3622's own
comment called for updating branch protection; that step never happened.

Fix keeps the contract in CODE rather than in protection settings: a
tiny aggregator job carrying the stable name, gated on the matrix's
aggregate result. A future shard-count change therefore cannot deadlock
the repo again. `if: always()` + result inspection so a legitimately
skipped matrix (filter says no core paths changed) still satisfies the
gate.

Verified: workflow YAML parses; job graph has dogfood-gate needs:
dogfood with name "Dogfood Regression Gate".

Co-authored-by: Jack Zhuang <277994282+os-zhuang@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jul 27, 2026
Merged
Closed
os-zhuang added a commit that referenced this pull request Jul 27, 2026
@os-zhuang @claude
chore(ci): cache Release's build + console dist, stop PR-side cache c…
b04b458 
…hurn, move CodeQL/coverage off the hot path (#3666)

Data from today's runs (post-#3622 steady state): every main push paid a
~9.5-minute Release job of which Build (4m21s, no turbo cache — the same
omission lint.yml had) and the vendored Console SPA build (3m11s, a pure
function of the .objectui-sha pin that rarely changes) were 80%. Meanwhile
PR-side cache saves (~5 turbo entries per push) churned the 10 GB Actions
cache pool and evicted the main seeds — observed as sudden cold spikes
(Build Core 51s → 4m30s).

- release.yml: add the standard turbo cache (Build ~4½min → expected <1min),
  and cache packages/console/dist keyed on hashFiles(.objectui-sha,
  build-console.sh) — skip build-console.sh on hit, and verify the dist
  stamp against the pin either way via check:console-sha (its exit-1-on-
  drift contract covers a stale restore).
- ci.yml + lint.yml: turbo caches become restore-only on PRs
  (actions/cache/restore); explicit main-push-only actions/cache/save steps
  (always(), matching the old post-step semantics) keep seeding. PRs read
  main's entries via the existing prefix restore-keys.
- codeql.yml: drop the pull_request trigger (user decision) — ~4½ runner-
  minutes per PR sync; every merge is analyzed on main within minutes at
  the current merge cadence, plus the weekly deep pass. Note left about the
  Analyze check no longer reporting on PRs.
- coverage → nightly (user decision): new coverage-nightly.yml (05:00 UTC +
  workflow_dispatch) owns the instrumented spec suite; ci.yml's push run now
  includes spec's plain test task instead (uninstrumented — net faster).
  Verified locally that spec's suite runs coverage without any build (spec
  has no workspace deps) and the v8 provider is present.

All five workflow files parse; changeset is empty (releases nothing).


Claude-Session: https://claude.ai/code/session_01ECTCrcCdZpCHw5zFSgcmGt

Co-authored-by: Claude <noreply@anthropic.com>
os-zhuang pushed a commit that referenced this pull request Jul 27, 2026
@claude
fix(ci): dogfood gate passes cancelled runs through — a cancelled mat…
e2efcb6 
…rix is not a failed one

Closes #3668. With cancel-in-progress on, every consecutive push cancelled
the in-flight dogfood matrix (the longest job in the workflow) and the
gate's catch-all branch turned that into a red X on the superseded SHA —
two observed on #3660 alone. False reds train everyone to ignore the one
check that must never be ignored.

Safety premise verified experimentally before landing (per the issue's own
ask): run 30271824408 executed a fail-fast matrix where shard 1 really
failed and fail-fast cancelled shard 2 mid-run — the aggregate
needs.<job>.result reads 'failure', not 'cancelled'. Failure dominates, so
an aggregate of 'cancelled' can only come from the whole run being stopped
externally (supersession or manual cancel) and passing it masks no real
regression. The manual-cancel case going green is the issue's accepted
trade-off; the rejected alternative (skipping the gate via !cancelled())
would republish the #3622 required-context deadlock.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ECTCrcCdZpCHw5zFSgcmGt
@os-zhuang os-zhuang mentioned this pull request Jul 27, 2026
Merged
os-zhuang added a commit that referenced this pull request Jul 27, 2026
@os-zhuang @claude
fix(ci): dogfood gate passes cancelled runs through — a cancelled mat…
c2ecfda 
…rix is not a failed one (#3671)

Closes #3668. With cancel-in-progress on, every consecutive push cancelled
the in-flight dogfood matrix (the longest job in the workflow) and the
gate's catch-all branch turned that into a red X on the superseded SHA —
two observed on #3660 alone. False reds train everyone to ignore the one
check that must never be ignored.

Safety premise verified experimentally before landing (per the issue's own
ask): run 30271824408 executed a fail-fast matrix where shard 1 really
failed and fail-fast cancelled shard 2 mid-run — the aggregate
needs.<job>.result reads 'failure', not 'cancelled'. Failure dominates, so
an aggregate of 'cancelled' can only come from the whole run being stopped
externally (supersession or manual cancel) and passing it masks no real
regression. The manual-cancel case going green is the issue's accepted
trade-off; the rejected alternative (skipping the gate via !cancelled())
would republish the #3622 required-context deadlock.


Claude-Session: https://claude.ai/code/session_01ECTCrcCdZpCHw5zFSgcmGt

Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

ci/cd documentation Improvements or additions to documentation size/s tooling

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@os-zhuang @claude