Skip to content

docs(design): CI build performance roadmap + implementer notes (#363) - #364

Draft
scottschreckengaust wants to merge 2 commits into
mainfrom
docs/363-ci-build-performance
Draft

docs(design): CI build performance roadmap + implementer notes (#363)#364
scottschreckengaust wants to merge 2 commits into
mainfrom
docs/363-ci-build-performance

Conversation

@scottschreckengaust

@scottschreckengaust scottschreckengaust commented Jun 16, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

Documents the CI build-performance roadmap in docs/design/CI_BUILD_PERFORMANCE.md (with its generated Starlight mirror). This satisfies the "keep CI_BUILD_PERFORMANCE.md current" DoD bullet of umbrella issue #363 — it is docs-only and implements no perf lever.

Relates to #363

#363 is an umbrella issue and stays OPEN after this docs PR merges. The remaining performance levers are tracked as separate ADR-003 child issues (see below), not implemented here.

What the doc covers

Doc claims were re-verified against current main after rebase: runner resolution from vars.DEFAULT_RUNNER_LABEL + PR labels, build runs on both pull_request and merge_group, contracts/coverage-thresholds.json + check:coverage-thresholds-sync drift check, and isolatedModules: true in cdk/tsconfig.jest.json — all still accurate.

Docs-only — perf levers are tracked as child issues

The four remaining levers are not implemented in this PR. They are proposed as ADR-003 child issues of #363 (for the orchestrator to open):

  1. Shard the CDK suite (jest --shard=N/M) — fan-out matrix + single aggregate build gate; cross-shard coverage merge before threshold enforcement.
  2. Gate coverage to merge_group — skip collectCoverage on pull_request push; keep thresholds enforced on merge_group.
  3. Bump the default runner (4-core → 8/16-core) — via vars.DEFAULT_RUNNER_LABEL; A/B against sharding.
  4. Path-filtered builds — make //cdk:test conditional (e.g. dorny/paths-filter) so docs/CLI/agent-only PRs skip it, while build still reports.

Testing

  • mise //docs:sync — no drift (committed Starlight mirror matches the regenerated output).
  • pre-commit run --files ... — docs-sync Passed, astro check Passed, gitleaks Passed.
  • mise //docs:build — full Astro/Starlight build Complete (70 pages).
  • Rebased onto origin/main (was 70 commits behind); clean, no conflicts. Scoped gitleaks on origin/main..HEAD — no leaks (the 3 full-history findings are pre-existing aws-account-id false positives in docs/backlog/* / docs/diagrams/*, unrelated to this diff).

🤖 Generated with Claude Code

@scottschreckengaust
scottschreckengaust requested review from a team as code owners June 16, 2026 22:08
@scottschreckengaust scottschreckengaust mentioned this pull request Jun 16, 2026
Open
3 tasks
@scottschreckengaust
scottschreckengaust marked this pull request as draft June 16, 2026 22:20
This was referenced Jun 16, 2026
Open
Closed
Merged
scottschreckengaust added a commit that referenced this pull request Jun 17, 2026
@scottschreckengaust @claude
docs(test): drop dead CI_BUILD_PERFORMANCE.md links (#366)
95ebf79 
The doc lives in draft PR #364 and exists on neither this branch nor
main, so the three links (disable-bundling.ts, AGENTS.md, review_pr.md)
are dead. Drop them and keep the stable #366 reference, removing the
merge-order dependency on #364 and the link-checker (#300) risk.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
isadeks pushed a commit to isadeks/sample-autonomous-cloud-coding-agents that referenced this pull request Jun 18, 2026
@scottschreckengaust @claude
perf(test): disable Lambda bundling in CDK unit-test synths (~15x per…
17eba2f 
… synth) (aws-samples#366) (aws-samples#371)

* perf(test): disable Lambda bundling in CDK unit-test synths (aws-samples#366)

Template.fromStack() triggers a full CDK synth that bundles every
NodejsFunction via esbuild — ~28s for the 413-resource AgentStack. Unit
tests assert on CloudFormation structure, not bundled Lambda code, so the
bundling is pure overhead. A jest setupFiles hook sets
aws:cdk:bundling-stacks: [] via CDK_CONTEXT_JSON, which a bare new App()
picks up with no call-site changes.

Measured: single AgentStack synth 28.7s -> 1.9s (~15x); github-tags.test.ts
~135s -> 9s; full //cdk:test 155s -> 25s (8-core local). All 2177 tests
pass; coverage thresholds unchanged (lines 92.36 >= 91).

Durable regression guards added: AGENTS.md "Common mistakes" entry and a
.abca/commands/review_pr.md checklist item — do not re-enable bundling in
tests unless asserting on a bundled asset, and minimize full-stack synths
(synth once in beforeAll). Both link docs/design/CI_BUILD_PERFORMANCE.md
(PR aws-samples#364).

Refs aws-samples#366. Part of aws-samples#363. Likely supersedes the sharding approach (aws-samples#365) —
to be re-evaluated once this lands.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(test): correct bundling opt-out to postCliContext (aws-samples#366)

The documented per-test opt-out used constructor `context`, which CDK
silently overwrites with CDK_CONTEXT_JSON in App.loadContext — so
`new App({ context: { 'aws:cdk:bundling-stacks': ['**'] } })` does NOT
re-enable bundling. Only `postCliContext` (applied last) overrides the
global disable.

Corrects the opt-out instruction and precedence comment in all three
documented locations (disable-bundling.ts, AGENTS.md, review_pr.md).
Verified empirically against aws-cdk-lib@2.257.0. Comment/doc-only;
runtime logic unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(test): clarify bundling opt-out is for asset output, not synth (aws-samples#366)

Disabling bundling does not stop tests from synthesizing —
`Template.fromStack()` still runs a full synth; it only skips the
esbuild asset-bundling step. The `postCliContext` opt-out exists solely
for the rare test that needs real bundled-asset output (asset hash /
S3 key), where an unbundled synth would silently yield a placeholder.
Makes this explicit in disable-bundling.ts and AGENTS.md. Doc-only.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(test): drop dead CI_BUILD_PERFORMANCE.md links (aws-samples#366)

The doc lives in draft PR aws-samples#364 and exists on neither this branch nor
main, so the three links (disable-bundling.ts, AGENTS.md, review_pr.md)
are dead. Drop them and keep the stable aws-samples#366 reference, removing the
merge-order dependency on aws-samples#364 and the link-checker (aws-samples#300) risk.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(test): harden bundling-disable against ambient CDK_CONTEXT_JSON (aws-samples#366)

Two robustness fixes to the global bundling-disable setup file, both from
PR aws-samples#371 review (finding 1 hardening nits):

- Spread BUNDLING_DISABLED_CONTEXT LAST when merging a pre-set
  CDK_CONTEXT_JSON, so an ambient `aws:cdk:bundling-stacks` value can no
  longer silently win and re-enable bundling. This is safe precisely because
  the documented per-test opt-out is `postCliContext` (applied after the env
  var regardless), so the escape hatch still works.
- Guard `JSON.parse` of the pre-set var with try/catch. A malformed value
  previously threw inside setupFiles and failed every test in every file with
  an opaque parse error; now the blame is localized to the offending value.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* test(cdk): add executable guard that Lambda bundling is disabled (aws-samples#366)

PR aws-samples#371 review finding 1: the ~15× synth speedup rested entirely on the
`disable-bundling.ts` setupFiles entry, but the only regression guards were
honor-system prose (AGENTS.md) and a review_pr.md checklist item. Dropping the
setupFiles wiring, reordering Jest setup, or a CDK rename of the context key
would silently revert the suite to full-bundling synths with no failing check.

Adds two assertions: the synth context carries `aws:cdk:bundling-stacks: []`,
and a bare-App stack reports `bundlingRequired === false`. Verified empirically
against aws-cdk-lib@2.259.0 that without the disable a bare App yields
`bundlingRequired === true`, so these discriminate a real regression.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* test(cdk): cover the postCliContext bundling opt-out precedence (aws-samples#366)

PR aws-samples#371 review finding 2: the documented per-test opt-out (`postCliContext`)
and the `loadContext` precedence this PR corrected had zero test coverage, so a
future CDK version reordering context application could silently break the one
supported escape hatch.

Locks the contract with two assertions verified against aws-cdk-lib@2.259.0:
`postCliContext: { 'aws:cdk:bundling-stacks': ['**'] }` re-enables bundling
(`bundlingRequired === true`), while constructor `context` does NOT (the env
var set by the global disable clobbers it). Uses a bare `Stack` rather than a
heavy AgentStack since the precedence is a property of CDK context loading, not
of any particular stack.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(test): reframe bundling as overhead-plus-smoke-test, not "pure overhead" (aws-samples#366)

PR aws-samples#371 review finding 3: "pure overhead" undersold the tradeoff. esbuild
bundling also validates that every Lambda `entry` resolves and the handler
compiles, so disabling it in unit tests means a broken entry path or handler TS
error stops surfacing at `Template.fromStack()` and only fails at real
synth/deploy or in the `agent/` build. The tradeoff is still acceptable — real
synth/deploy bundles, and handler code has its own compile/test under `agent/`
— but the wording now names what unit tests cede. Updated in both places the
phrase lived: the disable-bundling.ts header and AGENTS.md.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: bgagent <345885+scottschreckengaust@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
isadeks pushed a commit to isadeks/sample-autonomous-cloud-coding-agents that referenced this pull request Jul 28, 2026
@scottschreckengaust @claude
perf(test): disable Lambda bundling in CDK unit-test synths (~15x per…
ec8cf33 
… synth) (aws-samples#366) (aws-samples#371)

* perf(test): disable Lambda bundling in CDK unit-test synths (aws-samples#366)

Template.fromStack() triggers a full CDK synth that bundles every
NodejsFunction via esbuild — ~28s for the 413-resource AgentStack. Unit
tests assert on CloudFormation structure, not bundled Lambda code, so the
bundling is pure overhead. A jest setupFiles hook sets
aws:cdk:bundling-stacks: [] via CDK_CONTEXT_JSON, which a bare new App()
picks up with no call-site changes.

Measured: single AgentStack synth 28.7s -> 1.9s (~15x); github-tags.test.ts
~135s -> 9s; full //cdk:test 155s -> 25s (8-core local). All 2177 tests
pass; coverage thresholds unchanged (lines 92.36 >= 91).

Durable regression guards added: AGENTS.md "Common mistakes" entry and a
.abca/commands/review_pr.md checklist item — do not re-enable bundling in
tests unless asserting on a bundled asset, and minimize full-stack synths
(synth once in beforeAll). Both link docs/design/CI_BUILD_PERFORMANCE.md
(PR aws-samples#364).

Refs aws-samples#366. Part of aws-samples#363. Likely supersedes the sharding approach (aws-samples#365) —
to be re-evaluated once this lands.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(test): correct bundling opt-out to postCliContext (aws-samples#366)

The documented per-test opt-out used constructor `context`, which CDK
silently overwrites with CDK_CONTEXT_JSON in App.loadContext — so
`new App({ context: { 'aws:cdk:bundling-stacks': ['**'] } })` does NOT
re-enable bundling. Only `postCliContext` (applied last) overrides the
global disable.

Corrects the opt-out instruction and precedence comment in all three
documented locations (disable-bundling.ts, AGENTS.md, review_pr.md).
Verified empirically against aws-cdk-lib@2.257.0. Comment/doc-only;
runtime logic unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(test): clarify bundling opt-out is for asset output, not synth (aws-samples#366)

Disabling bundling does not stop tests from synthesizing —
`Template.fromStack()` still runs a full synth; it only skips the
esbuild asset-bundling step. The `postCliContext` opt-out exists solely
for the rare test that needs real bundled-asset output (asset hash /
S3 key), where an unbundled synth would silently yield a placeholder.
Makes this explicit in disable-bundling.ts and AGENTS.md. Doc-only.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(test): drop dead CI_BUILD_PERFORMANCE.md links (aws-samples#366)

The doc lives in draft PR aws-samples#364 and exists on neither this branch nor
main, so the three links (disable-bundling.ts, AGENTS.md, review_pr.md)
are dead. Drop them and keep the stable aws-samples#366 reference, removing the
merge-order dependency on aws-samples#364 and the link-checker (aws-samples#300) risk.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(test): harden bundling-disable against ambient CDK_CONTEXT_JSON (aws-samples#366)

Two robustness fixes to the global bundling-disable setup file, both from
PR aws-samples#371 review (finding 1 hardening nits):

- Spread BUNDLING_DISABLED_CONTEXT LAST when merging a pre-set
  CDK_CONTEXT_JSON, so an ambient `aws:cdk:bundling-stacks` value can no
  longer silently win and re-enable bundling. This is safe precisely because
  the documented per-test opt-out is `postCliContext` (applied after the env
  var regardless), so the escape hatch still works.
- Guard `JSON.parse` of the pre-set var with try/catch. A malformed value
  previously threw inside setupFiles and failed every test in every file with
  an opaque parse error; now the blame is localized to the offending value.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* test(cdk): add executable guard that Lambda bundling is disabled (aws-samples#366)

PR aws-samples#371 review finding 1: the ~15× synth speedup rested entirely on the
`disable-bundling.ts` setupFiles entry, but the only regression guards were
honor-system prose (AGENTS.md) and a review_pr.md checklist item. Dropping the
setupFiles wiring, reordering Jest setup, or a CDK rename of the context key
would silently revert the suite to full-bundling synths with no failing check.

Adds two assertions: the synth context carries `aws:cdk:bundling-stacks: []`,
and a bare-App stack reports `bundlingRequired === false`. Verified empirically
against aws-cdk-lib@2.259.0 that without the disable a bare App yields
`bundlingRequired === true`, so these discriminate a real regression.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* test(cdk): cover the postCliContext bundling opt-out precedence (aws-samples#366)

PR aws-samples#371 review finding 2: the documented per-test opt-out (`postCliContext`)
and the `loadContext` precedence this PR corrected had zero test coverage, so a
future CDK version reordering context application could silently break the one
supported escape hatch.

Locks the contract with two assertions verified against aws-cdk-lib@2.259.0:
`postCliContext: { 'aws:cdk:bundling-stacks': ['**'] }` re-enables bundling
(`bundlingRequired === true`), while constructor `context` does NOT (the env
var set by the global disable clobbers it). Uses a bare `Stack` rather than a
heavy AgentStack since the precedence is a property of CDK context loading, not
of any particular stack.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(test): reframe bundling as overhead-plus-smoke-test, not "pure overhead" (aws-samples#366)

PR aws-samples#371 review finding 3: "pure overhead" undersold the tradeoff. esbuild
bundling also validates that every Lambda `entry` resolves and the handler
compiles, so disabling it in unit tests means a broken entry path or handler TS
error stops surfacing at `Template.fromStack()` and only fails at real
synth/deploy or in the `agent/` build. The tradeoff is still acceptable — real
synth/deploy bundles, and handler code has its own compile/test under `agent/`
— but the wording now names what unit tests cede. Updated in both places the
phrase lived: the disable-bundling.ts header and AGENTS.md.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: bgagent <345885+scottschreckengaust@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
@scottschreckengaust @claude
docs(design): add CI build performance design doc (#363)
4175f85 
Records how the build workflow's cost is distributed (//cdk:test was the
~91% long pole), why it was expensive (redundant type-check, fixed in #357
/ PR #359), and the remaining levers (shard, coverage-gate, runner size,
path-filter) with sequence and implementer notes — including the
merge-queue required-check constraints that shape #2 and #5.

Companion to umbrella issue #363; keeps the optimization roadmap as a
durable, reviewable design artifact.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@scottschreckengaust
scottschreckengaust force-pushed the docs/363-ci-build-performance branch from 717a0af to 06a1f09 Compare July 29, 2026 00:51
@scottschreckengaust @claude
docs(design): add sharding design for CDK suite (#363)
d32c129 
Expands the CI build-performance doc with a full design for item #2
(shard the CDK suite): the jest --shard mechanism, a shards-vs-wall-time
table showing fixed overhead dominates past ~4 shards, the fan-out +
aggregate-gate pattern needed to keep `build` a single required context
on merge_group, cross-shard coverage merge before threshold enforcement,
and open questions (synth placement, deploy artifact, shard balance).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@scottschreckengaust
scottschreckengaust force-pushed the docs/363-ci-build-performance branch from 06a1f09 to d32c129 Compare July 29, 2026 00:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@scottschreckengaust