Skip to content

Rebuild around build-time aggregation (was broken by CORS)#1

Merged
widgetii merged 1 commit into
mainfrom
rebuild/build-time-aggregator
Jun 4, 2026
Merged

Rebuild around build-time aggregation (was broken by CORS)#1
widgetii merged 1 commit into
mainfrom
rebuild/build-time-aggregator

Conversation

@widgetii
Copy link
Copy Markdown
Member

@widgetii widgetii commented Jun 4, 2026

Summary

v0.1 fetched `sizes..json` by URL at runtime. Those URLs in
`manifest.json[builds[].platforms..sizes.url]` pointed at GitHub
release assets, which return no `Access-Control-Allow-Origin` header.
Every browser-side fetch was blocked by CORS. The original test plan
"verified CORS" with `curl -sfL`, which doesn't enforce CORS — the
site looked fine in CI and broke in real browsers.

This rewrite removes the runtime cross-origin fetch entirely. The
explorer is now a build-time aggregator: `scripts/prebuild.mts` walks
the last 90 nightly tags of OpenIPC/firmware and OpenIPC/builder via the
`gh` CLI (server-side, CORS-free), downloads every `sizes.*.json`
asset, and writes per-platform shards plus a small catalogue under
`public/data/`. Vite copies `public/` into `dist/`. The deployed
site fetches only relative `./data/...` URLs at runtime — same-origin
by construction.

Launch byte budget

Resource Gzipped
HTML + JS + CSS ~57 KB
`data//index.json` bootstrap ~3 KB
Empty-state landing total ~60 KB
One platform picked ~5 KB
Drift comparison ~5 KB

Tests (none existed in v0.1 — that's how this bug shipped)

  • `tests/prebuild.test.ts` (8) — `runPrebuild` driven by in-memory
    fs + canned `gh` CLI; retention trim, body parsing, simple+compound
    platform names, asset filtering.
  • `tests/lib.test.ts` (25) — categorise rules, drift diff
    (added/removed/changed, abs-delta sort), URL query round-trip.
  • `tests/bundle.test.ts` (5) — production `dist/` invariants.
    Most importantly asserts JS chunks do NOT embed
    `release-assets.githubusercontent.com`,
    `api.github.com/.../assets`, or `releases/download/*.json` —
    exactly the regression that would catch a repeat of the v0.1 bug.
    Also enforces a 250 KB bundle perf budget.
  • `tests/live.test.ts` (3, opt-in via `RUN_LIVE_TESTS=1`)
    post-deploy smoke that hits the live Pages URL, asserts
    `index.json` is reachable, a per-platform shard loads, and every
    URL is same-origin under `/firmware-explorer/`.

All 38 active tests pass locally. 3 live tests skipped (will run after
deploy via the `workflow_dispatch` path).

CI (`.github/workflows/pages.yml`)

  • `test` job (`npm ci && npm test`) gates `build`.
  • `schedule: cron '30 4 * * *'` — daily, after firmware (~00:11)
    and builder (~07:20) nightlies finish.
  • `build` job runs `npm run build`, which invokes `npm run prebuild`
    via npm lifecycle. Pulls release assets server-side using
    `${{ github.token }}`.

Upstream coordination

Both predicate reverts merged:

Out of scope / roadmap

  • WhatIfPanel stub deleted. The real configurator (Kconfig graph, dep
    resolution, defconfig fragment download) lands in v0.2 once
    OpenIPC/firmware emits `kconfig-graph..json` next to
    `sizes..json`.
  • Submit-to-builder trigger remains v0.3.

Test plan

  • Local: `npm test` — 38 tests pass, 3 live skipped.
  • Local: `npm run prebuild` against real OpenIPC/firmware releases — 97/97 platforms downloaded for `nightly-20260604-679c2c7`.
  • Local: `npm run build` — `dist/` 3.6 MB, JS 166 KB raw / 53 KB gzipped.
  • Local: `grep -rE 'release-assets|releases/download/.+\.json' dist/assets/*.js` — no matches.
  • CI test job green on this branch.
  • After merge: cron-triggered build + deploy populates Pages.
  • Post-deploy: `RUN_LIVE_TESTS=1 npm test` from a clean checkout, expect 3 live tests green.

🤖 Generated with Claude Code

v0.1 fetched sizes.<plat>.json by URL at runtime. The URLs in
manifest.json[builds[].platforms.<plat>.sizes.url] pointed at GitHub
release assets, which return no Access-Control-Allow-Origin header.
Every browser-side fetch was blocked by CORS. The original test plan
"verified CORS" with `curl -sfL`, which doesn't enforce CORS — the
site looked fine in CI and broke in real browsers.

This rewrite removes the runtime cross-origin fetch entirely. The
explorer is now a build-time aggregator:

  scripts/prebuild.mts walks the last 90 nightly tags of
  OpenIPC/firmware and OpenIPC/builder via the gh CLI (server-side,
  CORS-free), downloads every sizes.*.json asset, and writes:

    public/data/<source>/<tag>/sizes.<plat>.json   (per-platform shard)
    public/data/<source>/index.json                (catalogue)

  Vite copies public/ into dist/. The deployed site fetches only
  relative URLs (./data/...) at runtime — same-origin by construction,
  no CORS gate to fail.

Launch byte budget:
  - bundle (html+js+css):              ~57 KB gzipped
  - data/<source>/index.json bootstrap:  ~3 KB gzipped
  - empty-state landing total:         ~60 KB gzipped
  - per platform picked:               ~5 KB gzipped
  - drift comparison:                 +~5 KB gzipped

dist/ holds many small shard files (~200 platforms × ~90 builds × ~30 KB
each), but those are the explorer's own deliverable, not data piled
into firmware/builder's gh-pages.

Tests (none in v0.1 — that's how this bug shipped):
  - tests/prebuild.test.ts (8): runPrebuild with in-memory fs and
    canned gh CLI driver; retention, body parsing, simple/compound
    platform names, asset filtering.
  - tests/lib.test.ts (25): categorise rules, drift diff (added /
    removed / changed, abs-delta sort), URL query round-trip.
  - tests/bundle.test.ts (5): production dist invariants. Most
    importantly, asserts the JS chunks do NOT embed
    release-assets.githubusercontent.com, api.github.com/.../assets,
    or releases/download/*.json — exactly the regression that would
    catch a repeat of the v0.1 bug. Also enforces a 250 KB bundle
    perf budget.
  - tests/live.test.ts (3, opt-in via RUN_LIVE_TESTS=1): post-deploy
    smoke that hits the live Pages URL, asserts index.json is reachable,
    a per-platform shard loads, and every URL is same-origin under
    /firmware-explorer/.

CI changes (.github/workflows/pages.yml):
  - test job (npm ci && npm test) gates build job
  - schedule: cron daily 04:30 UTC, after firmware/builder nightlies
  - build job runs `npm run build` (which invokes `npm run prebuild`
    via npm lifecycle), pulls release assets server-side using the
    workflow's GH_TOKEN.

Sources of upstream change (must land first):
  - OpenIPC/firmware#2169 (manifest revert)
  - OpenIPC/builder#103   (manifest revert)
  Both merged.

Out of scope, kept on the roadmap:
  - WhatIfPanel stub deleted; the real configurator (Kconfig graph,
    dep resolution, defconfig fragment download) lands in v0.2 once
    OpenIPC/firmware emits kconfig-graph.<plat>.json next to
    sizes.<plat>.json.
  - submit-to-builder trigger remains v0.3.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@widgetii widgetii merged commit e1971e3 into main Jun 4, 2026
2 of 3 checks passed
@widgetii widgetii deleted the rebuild/build-time-aggregator branch June 4, 2026 12:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant