Rebuild around build-time aggregation (was broken by CORS)

.github/workflows/pages.yml

@@ -3,6 +3,10 @@ name: Deploy to GitHub Pages
 on:
   push:
     branches: [main]
+  schedule:
+    # 04:30 UTC daily, well after firmware (00:11) and builder (~07:20)
+    # nightlies have settled. Cron format: minute hour dom month dow.
+    - cron: "30 4 * * *"
   workflow_dispatch:
 
 permissions:
@@ -15,7 +19,19 @@ concurrency:
   cancel-in-progress: true
 
 jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: "22"
+          cache: npm
+      - run: npm ci
+      - run: npm test
+
   build:
+    needs: test
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
@@ -24,7 +40,14 @@ jobs:
           node-version: "22"
           cache: npm
       - run: npm ci
+      # `npm run build` invokes `npm run prebuild` first, which uses the
+      # built-in `gh` CLI to download `sizes.*.json` release assets from
+      # OpenIPC/firmware and OpenIPC/builder, server-side. CORS is structurally
+      # not a question during this step — only the in-browser fetches need to
+      # stay same-origin.
       - run: npm run build
+        env:
+          GH_TOKEN: ${{ github.token }}
       - uses: actions/configure-pages@v5
       - uses: actions/upload-pages-artifact@v3
         with:

.gitignore

@@ -4,3 +4,7 @@ dist
 *.log
 .DS_Store
 *.tsbuildinfo
+# Generated by `npm run prebuild` from upstream release assets; rebuilt
+# nightly by the Pages workflow. Not source — kept out of git.
+public/data/
+coverage/

README.md

@@ -1,59 +1,105 @@
 # firmware-explorer
 
-Per-build package + kernel-module composition explorer for OpenIPC firmware.
+Per-build package + kernel-module composition viewer for OpenIPC firmware.
 
 Live: **https://openipc.github.io/firmware-explorer/**
 
 ## What it does
 
-Reads the `manifest.json` published at:
-
-- https://openipc.github.io/firmware/manifest.json
-- https://openipc.github.io/builder/manifest.json
-
-…and, for every build × platform that ships a `sizes.<plat>.json` sidecar (from `OpenIPC/firmware` PR [#2166](https://github.com/OpenIPC/firmware/pull/2166) and `OpenIPC/builder` PR [#102](https://github.com/OpenIPC/builder/pull/102)), renders:
+For each nightly build × platform that ships a `sizes.<plat>.json` sidecar (from `OpenIPC/firmware` and `OpenIPC/builder` release assets), it renders:
 
 - **Size summary** — kernel/uImage and rootfs cap utilisation with a red/amber/green badge.
 - **Treemap** — packages sized by uncompressed bytes, coloured by category (vendor SDK / kernel / userspace / toolchain / overlay).
 - **Package table** — sortable by name / uncompressed / compressed-approx / file count; click a row to expand `top_files[]`.
 - **Module table** — every `.ko` with its owning package and an "autoloaded by `/etc/modules`" flag; filter to "on-demand only" to surface compaction candidates.
 - **Removed-by-finalize panel** — files the build dropped via OpenIPC-specific finalize hooks (`HISILICON_OPENSDK_TRIM_SP2308`, the libstdc++ strip from `rootfs_script.sh`, per-board excludes lists).
-- **What-if scratchpad** — tick boxes to mark packages / modules as "would disable", see a running total of the rootfs savings. **Naive sum, no dependency resolution yet** (see [v0.2 below](#roadmap)).
-- **Drift view** — diff any two builds for the same platform, sorted by absolute byte delta. Catches the kind of week-on-week growth that PR [firmware#2163](https://github.com/OpenIPC/firmware/pull/2163) had to wait for a rootfs overflow to detect.
+- **Drift view** — diff any two builds for the same platform, sorted by absolute byte delta.
+
+Source toggle switches between OpenIPC/firmware nightlies and OpenIPC/builder per-device nightlies. Selection state is encoded in the URL query so views are shareable.
+
+## Architecture
+
+The explorer is a **build-time aggregator**, not a runtime fetcher. The browser never crosses an origin boundary at runtime.
+
+```
+GitHub release assets ──(server-side, no CORS)──> explorer CI prebuild
+                                                       │
+                                                       ▼
+                          public/data/<source>/<build>/sizes.<plat>.json
+                          public/data/<source>/index.json
+                                                       │
+                                            (Vite copies public/ to dist/)
+                                                       │
+                                                       ▼
+                               gh-pages static site at openipc.github.io/firmware-explorer/
+                                                       │
+                                       (browser fetches same-origin only)
+                                                       ▼
+                                             user views build × platform
+```
+
+`scripts/prebuild.mts` is the workhorse: it walks each repo's nightly releases via the `gh` CLI, downloads every `sizes.*.json` asset server-side (CORS is structurally not a question there), and writes per-platform JSON shards plus a small `index.json` catalogue under `public/data/`. Vite copies the entire `public/` into `dist/`, the deploy workflow pushes that to GitHub Pages, and the runtime only ever fetches relative URLs.
 
-Source toggle lets you switch between the firmware repo's nightlies and the builder repo's (per-device) nightlies. Selection state is encoded in the URL query so views are shareable.
+### Why this matters
+
+Earlier versions of this tool used `manifest.json[builds[].platforms.<plat>.sizes.url]` URLs at runtime, which pointed at GitHub release assets. Release assets do not set `Access-Control-Allow-Origin`, so every browser fetch was blocked by CORS — the tool only appeared to work in `curl`. The build-time aggregator design removes the cross-origin fetch entirely. `tests/bundle.test.ts` enforces this by grepping the production JS for the killer URL patterns.
+
+## Launch byte budget
+
+| Resource | Gzipped |
+|---|---|
+| `index.html` + JS bundle + CSS | ~57 KB |
+| `data/<default source>/index.json` | ~3 KB |
+| **Empty-state landing total** | **~60 KB** |
+| One platform's JSON (after pick) | ~5 KB |
+| Drift comparison (one more shard) | ~5 KB |
 
 ## Local dev
 
+Requires Node 20+ and `gh` CLI authenticated (`gh auth login`) for the prebuild step.
+
 ```bash
 npm install
-npm run dev    # http://localhost:5173/
-npm run build  # → dist/
-npm run preview
+npm run dev          # http://localhost:5173/firmware-explorer/
+npm run build        # prebuild → tsc → vite build → dist/
+npm test             # vitest suite (38 tests; 3 live tests skipped)
+RUN_LIVE_TESTS=1 npm test
 ```
 
-Requires Node 20+. The dev server fetches manifests directly from `openipc.github.io` (CORS: allow-`*`), so no local data is needed.
+Prebuild knobs:
+
+| Env var | Default | Purpose |
+|---|---|---|
+| `RETENTION` | 90 | Number of nightlies to retain |
+| `LIMIT` | (none) | Cap on tags processed (debug; e.g. `LIMIT=3` for a fast iteration) |
+| `SOURCES` | `firmware,builder` | Comma-separated subset |
+| `FORCE_REFETCH` | (off) | Skip the on-disk download cache and re-download every tag |
+| `QUIET` | (off) | Suppress per-tag progress output |
+
+`gh release download` is cached under `/tmp/firmware-explorer-prebuild/` so repeated builds are fast.
 
 ## Deployment
 
-`.github/workflows/pages.yml` builds on every push to `main` and deploys `dist/` to GitHub Pages. The Vite `base` is set to `/firmware-explorer/` so asset paths resolve correctly under the gh-pages subpath.
+`.github/workflows/pages.yml` runs:
+
+1. `test` job — `npm ci && npm test`. Hard gate; deploys are blocked on red tests.
+2. `build` job — `npm ci && npm run build` (prebuild downloads release assets via the workflow's `GH_TOKEN`).
+3. `deploy` job — `actions/deploy-pages@v4` publishes `dist/` to gh-pages.
+
+Triggers: push to `main`, nightly cron at 04:30 UTC (after firmware ~00:11 and builder ~07:20 nightlies finish), and manual `workflow_dispatch`.
 
 ## Schema
 
-This explorer expects `schema: 1` on both `manifest.json` and `sizes.<plat>.json`. The authoritative emitter is [`OpenIPC/firmware`'s `general/scripts/size_report.py`](https://github.com/OpenIPC/firmware/blob/master/general/scripts/size_report.py); the explorer's [`src/lib/types.ts`](src/lib/types.ts) mirrors that schema. Any schema bump on the emitter side will require a matching change here.
+The explorer expects `schema: 1` on both `data/<source>/index.json` (defined in this repo's [`src/lib/types.ts`](src/lib/types.ts)) and the per-platform `sizes.<plat>.json` (defined upstream in [`OpenIPC/firmware`'s `general/scripts/size_report.py`](https://github.com/OpenIPC/firmware/blob/master/general/scripts/size_report.py)). A schema bump on the upstream emitter needs a matching change here.
 
 ## Roadmap
 
-This is **v0.1** — the viewer and the what-if scratchpad.
-
-- **v0.2** — real Kconfig dependency resolution. Two PRs:
-  - `OpenIPC/firmware`: new `general/scripts/kconfig_graph.py` that walks `general/openipc.fragment` + every `package/*/Config.in` and emits `kconfig-graph.<plat>.json` next to `sizes.<plat>.json`.
-  - This repo: consume it, gate impossible toggles, generate downloadable defconfig fragments ("paste these lines into your board defconfig").
-- **v0.3 (optional)** — submit a config to the builder pipeline and trigger a build. Needs hosted infrastructure.
+- **v0.2 — Kconfig-aware configurator** — `OpenIPC/firmware: general/scripts/kconfig_graph.py` emits a `kconfig-graph.<plat>.json` next to `sizes.<plat>.json`. This repo's prebuild aggregates it the same way as sizes; the explorer gains dep-aware toggles and a downloadable `# BR2_PACKAGE_FOO is not set` defconfig fragment. Deferred — needs the firmware-side emitter to land first.
+- **v0.3 (optional) — submit-to-builder trigger** — needs hosted infrastructure; only worth doing once v0.2 proves the workflow useful.
 
 ## Issue trail
 
-Originally requested in [OpenIPC/firmware#242](https://github.com/OpenIPC/firmware/issues/242) (in Russian: "package tree for a firmware web configurator"). The data side shipped in [`OpenIPC/firmware#2166`](https://github.com/OpenIPC/firmware/pull/2166) + [`OpenIPC/builder#102`](https://github.com/OpenIPC/builder/pull/102); this repo is the consumer.
+Originally requested in [OpenIPC/firmware#242](https://github.com/OpenIPC/firmware/issues/242). Size-report data is emitted by [`OpenIPC/firmware#2166`](https://github.com/OpenIPC/firmware/pull/2166) + [`OpenIPC/builder#102`](https://github.com/OpenIPC/builder/pull/102). This repo is the build-time consumer.
 
 ## License