From 27db303f54c5e468fe93b3576a0cccc6bb93cf16 Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Tue, 14 Jul 2026 09:56:26 +0000 Subject: [PATCH 1/7] fix(ci): build the bundle before packing, and prove it is in the tarball MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CI ran `npx tsc` as its build step and then packed that tree. tsc emits the per-module dist; it does not run esbuild, so dist/krm-stream.js was never built — and release.yml publishes CI's exact bytes rather than rebuilding, by design. The next release would therefore have published a package whose exports map advertises "./bundle": { "default": "./dist/krm-stream.js" } pointing at a file that is not in the tarball. Every `import "@configbutler/krm-stream/bundle"` fails with ERR_MODULE_NOT_FOUND, in a release that passed CI green — because nothing in the pipeline imports the published artifact. The whole point of the ./bundle work, silently absent, plus a broken exports entry that is worse than not shipping it at all. Reproduced locally against the real sequence (npm ci → npx tsc → npm pack): the tarball contains dist/index.js and no dist/krm-stream.js. Two changes: - the build step is `npm run build`, which is tsc AND the esbuild flatten. One source of truth with package.json. - a guard after `npm pack` asserts that every entry point in the exports map is actually IN the tarball. A comment saying "keep these in step" rots; this fails the build. Verified it rejects the bundle-less tarball and passes the correct one. Co-Authored-By: Claude Opus 4.8 (1M context) --- .github/workflows/ci.yml | 29 ++++++++++++++++++++++++++++- 1 file changed, 28 insertions(+), 1 deletion(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 183e87e..dfaaf80 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -232,7 +232,14 @@ jobs: working-directory: packages/krm-stream # The build, last: it is what ships, and it must stay plain ESM a browser can import with no # bundler and no runtime dependency. - - run: npx tsc + # + # `npm run build`, NOT `npx tsc`. The build is two steps now — tsc emits the per-module dist, + # then esbuild flattens it into the single-file dist/krm-stream.js that package.json publishes as + # `./bundle`. `npx tsc` alone produces a tree that packs and publishes CLEANLY while missing the + # bundle entirely, and the exports map would then point `./bundle` at a file that is not in the + # tarball: every `import "@configbutler/krm-stream/bundle"` fails with ERR_MODULE_NOT_FOUND, in a + # release that passed CI. Keep this in step with package.json's build script. + - run: npm run build working-directory: packages/krm-stream # Pack the official client HERE, in the job that just tested it, and hand it to the release @@ -248,6 +255,26 @@ jobs: set -euo pipefail mkdir -p /tmp/npm npm pack --pack-destination /tmp/npm ./packages/krm-stream + + # Every entry point in the exports map must actually BE in the tarball. + # + # A missing file here is invisible in every other job: the build succeeds, the tests pass, the + # pack succeeds, and the package publishes. It fails for the first consumer who imports the entry + # point we advertised and gets ERR_MODULE_NOT_FOUND — at which point the fix is another release. + # release.yml publishes these bytes without rebuilding them, so this is the last place to look. + - name: the tarball contains every entry point package.json advertises + run: | + set -euo pipefail + tarball=$(ls /tmp/npm/configbutler-krm-stream-*.tgz) + for entry in package/dist/index.js package/dist/krm-stream.js; do + if ! tar -tzf "$tarball" | grep -qx "$entry"; then + echo "::error::${entry#package/} is in the exports map but NOT in the tarball — publishing this would ship a broken import." + tar -tzf "$tarball" + exit 1 + fi + done + echo "both entry points are present in $(basename "$tarball")." + - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: npm-packages From 5713504e569629f15854a84001d011667f77c064 Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Tue, 14 Jul 2026 10:03:34 +0000 Subject: [PATCH 2/7] fix(ci)!: build the bundle before packing, and make the caches stop lying MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three bugs, one disease: a step that cannot tell whether its own output exists. # 1. CI packed a bundle it never built CI built with `npx tsc` and packed that tree. tsc emits the per-module dist; it does not run esbuild, so dist/krm-stream.js was never built — and release.yml publishes CI's exact bytes rather than rebuilding, by design. The next release would have published a package whose exports map advertises "./bundle": "./dist/krm-stream.js" pointing at a file NOT IN THE TARBALL. Every `import "@configbutler/krm-stream/bundle"` fails with ERR_MODULE_NOT_FOUND, in a release that passed CI green — because nothing in the pipeline ever imports the published artifact. # 2. `task build-client` could not see its own outputs rm dist/krm-stream.js && task build-client → "Task is up to date" (nothing rebuilt) `generates: ["dist/**/*.js"]` is only a checksum key; Task never asks whether the files exist. Outputs are now named individually AND guarded by a `status:` that probes the filesystem, so a missing artifact rebuilds. # 3. npm install prunes node_modules across branches node_modules is shared mutable state that git does not track, and `npm install` reconciles it to the CURRENT branch's package.json. Running any task from a branch predating a devDependency DELETES it; switching back leaves a build dying on `sh: 1: esbuild: not found`, a message that points at nothing. _client-deps/_example-deps now use `npm ci` — the tree is a pure function of two committed files — fingerprinted on those files and status-guarded on the binaries actually being present. # One definition of a thing `task pack-client` builds, packs, and ASSERTS every entry point in the exports map is in the tarball. CI runs that same task instead of a hand-copied shell line, so the local gate and the release gate cannot drift again. `task verify` is the whole gate in CI's order. The guard caught a bug in itself on its first run: `tar | grep -q` under pipefail reports FAILURE on a match, because grep exits early and tar takes SIGPIPE. It lists once into a variable now. Verified in both directions — it passes a good tarball and rejects a tsc-only one. Co-Authored-By: Claude Opus 4.8 (1M context) --- .github/workflows/ci.yml | 60 ++++--------- Taskfile.yml | 178 ++++++++++++++++++++++++++++++++++++--- 2 files changed, 185 insertions(+), 53 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index dfaaf80..8f32ae3 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -217,6 +217,12 @@ jobs: node-version: "22" cache: npm cache-dependency-path: packages/krm-stream/package-lock.json + # task, so the build+pack step below is the SAME task a developer runs. The two used to be + # hand-copied shell lines in two files, and they drifted the moment the build grew a step. + - uses: arduino/setup-task@c0bc642852239c2689f73f4ea6459c29405f3c52 # v3.0.0 + with: + version: 3.x + repo-token: ${{ secrets.GITHUB_TOKEN }} # npm ci, not npm install: the lockfile is the input, and a build that is allowed to quietly # resolve a different tree than the one committed is not the build we tested. - run: npm ci --no-audit --no-fund @@ -230,50 +236,20 @@ jobs: working-directory: packages/krm-stream - run: npx biome ci working-directory: packages/krm-stream - # The build, last: it is what ships, and it must stay plain ESM a browser can import with no - # bundler and no runtime dependency. + # The build and the pack, as ONE task — the same one a developer runs as `task pack-client`. # - # `npm run build`, NOT `npx tsc`. The build is two steps now — tsc emits the per-module dist, - # then esbuild flattens it into the single-file dist/krm-stream.js that package.json publishes as - # `./bundle`. `npx tsc` alone produces a tree that packs and publishes CLEANLY while missing the - # bundle entirely, and the exports map would then point `./bundle` at a file that is not in the - # tarball: every `import "@configbutler/krm-stream/bundle"` fails with ERR_MODULE_NOT_FOUND, in a - # release that passed CI. Keep this in step with package.json's build script. - - run: npm run build - working-directory: packages/krm-stream - - # Pack the official client HERE, in the job that just tested it, and hand it to the release - # workflow as an artifact. release.yml publishes these exact bytes rather than rebuilding it: - # the things on npm are then the things this pipeline proved, and not a second build that - # merely ought to match. + # It builds (tsc AND the esbuild flatten — `npx tsc` alone silently produces a dist with no + # bundle), packs the tarball the release workflow will publish, and then ASSERTS that every entry + # point in the exports map is actually in that tarball. That last check is the one nothing else + # can do: a missing ./bundle builds green, tests green, packs green, publishes green, and fails + # for the first consumer who imports the entry point we advertised. # - # The versions inside them are already correct on a release commit — release-please bumps - # package.json in the release PR, so by the time this runs on main the tree IS the release. - - name: pack the tarballs that will be published - # mkdir first: --pack-destination does not create the directory, it fails with ENOENT. - run: | - set -euo pipefail - mkdir -p /tmp/npm - npm pack --pack-destination /tmp/npm ./packages/krm-stream - - # Every entry point in the exports map must actually BE in the tarball. - # - # A missing file here is invisible in every other job: the build succeeds, the tests pass, the - # pack succeeds, and the package publishes. It fails for the first consumer who imports the entry - # point we advertised and gets ERR_MODULE_NOT_FOUND — at which point the fix is another release. - # release.yml publishes these bytes without rebuilding them, so this is the last place to look. - - name: the tarball contains every entry point package.json advertises - run: | - set -euo pipefail - tarball=$(ls /tmp/npm/configbutler-krm-stream-*.tgz) - for entry in package/dist/index.js package/dist/krm-stream.js; do - if ! tar -tzf "$tarball" | grep -qx "$entry"; then - echo "::error::${entry#package/} is in the exports map but NOT in the tarball — publishing this would ship a broken import." - tar -tzf "$tarball" - exit 1 - fi - done - echo "both entry points are present in $(basename "$tarball")." + # These are the bytes release.yml publishes — it does not rebuild them, so the things on npm are + # the things this pipeline proved, and not a second build that merely ought to match. The + # versions inside are already correct on a release commit: release-please bumps package.json in + # the release PR, so by the time this runs on main the tree IS the release. + - name: build, pack, and prove the tarball is complete + run: task pack-client PACK_DIR=/tmp/npm - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: diff --git a/Taskfile.yml b/Taskfile.yml index ecb6b21..150ba07 100644 --- a/Taskfile.yml +++ b/Taskfile.yml @@ -1,7 +1,38 @@ version: "3" # Everything in this repository is driven by `task`. -# `task` with no argument lists what there is. +# `task` with no argument lists what there is; `task verify` is the whole gate. +# +# # A cache that lies is worse than no cache +# +# Task skips work by CHECKSUMMING `sources`. That is a statement about inputs, and it says nothing +# whatsoever about whether the outputs still exist. Two real failures came out of that gap, and both +# cost real time before they were understood: +# +# 1. `rm dist/krm-stream.js && task build-client` → "Task is up to date". Nothing rebuilt. The +# sources had not changed, so the fingerprint matched, so Task was — by its own rules — correct. +# A half-built dist survived a build. +# 2. node_modules is shared mutable state across git branches, and `npm install` PRUNES to whatever +# the current branch's package.json says. Run a task on a branch that predates a devDependency +# and it is silently deleted from the tree; switch back and the build dies on +# `sh: 1: esbuild: not found`, which points at nothing. +# +# Both are the same bug as the one in CI that this all came from — a pipeline that packed a bundle it +# had never built. So the rules here: +# +# - Every cache-worthy task DELIVERS A FILE, named individually in `generates` (never a glob, which +# is only a checksum key). +# - Every such task also carries a `status:` that PROBES THE FILESYSTEM. That is the check a +# checksum cannot do, and it is what makes a deleted artifact or a pruned dependency self-heal +# instead of being skipped over. +# - Dependencies are installed with `npm ci`, never `npm install`: the tree becomes a pure function +# of two committed files and cannot drift from the branch you are on. +# - One definition of a thing. The build is `npm run build` in package.json, and CI runs the same +# `task pack-client` a developer does. Every hand-copied shell line in a second file is a thing +# that will drift, and it did. +# +# Go's build/test cache is deliberately NOT fought with. It is content-addressed and sound: a cached +# `ok` means those exact inputs really did pass. `task clean` does not touch it. run: once @@ -58,6 +89,21 @@ tasks: cmds: - git diff --exit-code -- conformance/gen || (echo "conformance/gen is stale — run 'task fixtures' and commit" && exit 1) + # ------------------------------------------------------------------ verify -- + verify: + desc: "The whole gate, in the order CI runs it. If this passes, CI passes." + cmds: + # Sequential on purpose: this is the pre-push check, and the first failure is the one you want + # to read, not the fourth. `task test` and `task lint` fan out internally. + - task: fixtures-check + - task: lint + - task: test + - task: e2e-wire + - task: e2e-browser + # Last, because it is the one nothing else can prove: what we UPLOAD contains what the exports + # map advertises. Every rung above this builds its own dist and never looks at the tarball. + - task: pack-client + # -------------------------------------------------------------------- test -- test: desc: "Run both suites against the shared conformance fixtures." @@ -84,7 +130,7 @@ tasks: test-client: desc: "TypeScript: the client + its half of the conformance suite (node --test, no deps)." - deps: [fixtures] + deps: [fixtures, _client-deps] dir: "{{.CLIENT_DIR}}" cmds: - node --test @@ -128,18 +174,18 @@ tasks: cd {{.CLIENT_DIR}} && REPLAY_URL=http://{{.REPLAY_ADDR}} node e2e/wire.ts e2e-browser: - desc: "End to end in a REAL browser: native EventSource, unbundled ESM, no cluster." + desc: "End to end in a REAL browser: native EventSource, unbundled ESM AND the bundle, no cluster." # build-client, because playwright's webServer rebuilds the library from packages/krm-stream — # and it cannot do that with no node_modules there. On a developer's machine that directory is # already populated by `task test`, which is exactly why this gap survived until the first CI run # on a clean checkout found it. - deps: [fixtures, build-client] + deps: [fixtures, build-client, _example-deps] dir: examples/vanilla-browser cmds: # The only place the library's central promise is actually tested: that the published ESM # imports in a browser with no bundler. Node importing it proves nothing — Node is not a browser. - # Playwright lives in the EXAMPLE's package, so the library keeps its three devDependencies. - - npm install --no-audit --no-fund + # Playwright lives in the EXAMPLE's package, so the library keeps its four devDependencies. + # Runs the suite twice — once per entry point (per-module and ./bundle). - npx --no-install playwright install chromium --with-deps - npx --no-install playwright test {{.CLI_ARGS}} @@ -172,6 +218,7 @@ tasks: - golangci-lint run --build-tags e2e ./... lint-client: + deps: [_client-deps] dir: "{{.CLIENT_DIR}}" cmds: # Two tsconfigs, on purpose. The first is the BUILD (src only — its rootDir is what ships); @@ -185,27 +232,136 @@ tasks: fmt-client: desc: "Format and auto-fix the TypeScript (the gofmt of this side)." + deps: [_client-deps] dir: "{{.CLIENT_DIR}}" cmds: - npx --no-install biome check --write + # -------------------------------------------------------------------- deps -- + # node_modules is SHARED MUTABLE STATE ACROSS GIT BRANCHES, and nothing in git tracks it. That is + # the whole reason these two tasks exist, and the bug they close cost a real half-hour: + # + # `npm install` reconciles node_modules to the CURRENT branch's package.json — which means it + # PRUNES. Run any task that calls it while sitting on a branch that predates a new devDependency, + # switch back to main, and the dependency is simply gone. The next build fails with + # `sh: 1: esbuild: not found` — a message that says nothing about branches, and sends you looking + # for a bug in the build instead of a bug in your working tree. + # + # `npm ci` instead: it installs EXACTLY the lockfile, deleting node_modules first, so the tree is a + # pure function of two committed files and can never drift from the branch you are on. It is + # fingerprinted on those two files, so it re-runs when (and only when) you switch to a branch whose + # dependencies differ — which is precisely when the drift would otherwise appear. + # + # The `generates` file is npm's own record of what it installed. It is a real artifact, not a + # stamp we invented. + _client-deps: + internal: true + dir: "{{.CLIENT_DIR}}" + sources: ["package.json", "package-lock.json"] + generates: ["node_modules/.package-lock.json"] + status: + # sources/generates is a CHECKSUM — it does not look at the filesystem. Without this, an + # `rm -rf node_modules` is invisible to Task and the next build fails on a missing binary. + - test -x node_modules/.bin/tsc + - test -x node_modules/.bin/esbuild + - test -x node_modules/.bin/biome + cmds: + - npm ci --no-audit --no-fund + + _example-deps: + internal: true + dir: examples/vanilla-browser + sources: ["package.json", "package-lock.json"] + generates: ["node_modules/.package-lock.json"] + status: + - test -x node_modules/.bin/playwright + cmds: + - npm ci --no-audit --no-fund + # ------------------------------------------------------------------- build -- build-client: desc: "Emit the dependency-free ESM a browser can