Add sandboxed benchmark skill

.agents/skills/sandbox-bench/SKILL.md

@@ -0,0 +1,172 @@
+---
+name: sandbox-bench
+description: Run framework-tracker framework, app, package, and version benchmark matrices on identical Vercel Sandbox VMs, including Next.js stable/canary comparisons and cross-framework SSR, load, browser-rendering, build, or install measurements. Use when asked to run, compare, repeat, or validate framework-tracker benchmarks remotely, on Vercel Sandbox, across independent machines, or without tying up the local workstation.
+---
+
+# Run framework-tracker benchmarks on Vercel Sandbox
+
+Use the repository's `pnpm benchmark` CLI as the source of truth for benchmark
+selection and version overrides. Use this skill's runner only for remote
+provisioning, identical setup snapshots, independent VM replication,
+collection, and boot-level summaries.
+
+Run every command below from the framework-tracker repository root.
+
+## Start by checking configuration and active work
+
+Before launching resources, run:
+
+```sh
+node .agents/skills/sandbox-bench/scripts/config.mjs show
+node .agents/skills/sandbox-bench/scripts/status.mjs
+```
+
+If configuration reports `NOT CONFIGURED`, ask which Vercel **team** and
+**project** should own and bill the sandboxes. Never guess or commit them.
+Save the answer outside the repository:
+
+```sh
+node .agents/skills/sandbox-bench/scripts/config.mjs set \
+  team=<team-slug> project=<project-name>
+```
+
+The Vercel CLI session must have access to that scope. Verify failures with:
+
+```sh
+vercel whoami --scope <team-slug>
+vercel sandbox list --team <team-slug> --project <project-name>
+```
+
+Stop after a persistent authorization failure instead of retrying resource
+creation. Configuration is stored in
+`~/.config/framework-tracker-sandbox-bench/config.json`; results and snapshot
+IDs are stored in `~/.cache/framework-tracker-sandbox-bench/`.
+
+## Resolve the comparison matrix
+
+Pass any framework or package selector accepted by `pnpm benchmark`. Append
+`@<version>` to make one case use a specific framework version.
+
+- `next@stable` is normalized to `next@latest`.
+- Use `next@canary` for the current npm canary.
+- Prefer exact app selectors, such as `app-next-js@canary`, for app-only runtime
+  comparisons.
+- A framework selector such as `next@canary` may produce both starter and app
+  results, depending on which measurements those packages configure.
+- Each versioned target becomes a separate invocation so the local CLI's
+  one-framework-per-`--version` invariant remains intact.
+- Use the resolved `frameworkVersion` in raw results when reporting dist-tags.
+
+Inspect available local cases when uncertain:
+
+```sh
+pnpm benchmark --list
+```
+
+Always inspect a no-op plan before creating billable resources:
+
+```sh
+node .agents/skills/sandbox-bench/scripts/sandbox-bench.mjs \
+  app-next-js@canary app-tanstack-start-react \
+  --measurement ssrLoad,ssrRequestThroughput \
+  --vms 8 --dry-run
+```
+
+The source bundle contains tracked changes plus non-ignored untracked files, so
+the sandbox measures the current working tree. The runner prints a source
+fingerprint and snapshots one prepared checkout; every measurement VM boots
+from that snapshot. Browser measurements automatically add Google Chrome.
+
+## Launch the run
+
+Typical runtime comparison:
+
+```sh
+node .agents/skills/sandbox-bench/scripts/sandbox-bench.mjs \
+  app-next-js@canary app-tanstack-start-react \
+  --measurement \
+  ssrLoad,ssrRequestThroughput,clientSideRendered,serverSideRendered \
+  --vms 8 --runs 5 --label next-canary-vs-tanstack
+```
+
+Next.js stable versus canary:
+
+```sh
+node .agents/skills/sandbox-bench/scripts/sandbox-bench.mjs \
+  app-next-js@stable app-next-js@canary \
+  --measurement ssrLoad,ssrRequestThroughput \
+  --vms 8 --label next-stable-vs-canary
+```
+
+Useful options:
+
+- `--vms <n>` controls independent boots. Use one only for a smoke test, eight
+  for exploration, and sixteen when a small effect may drive a decision.
+- `--concurrency <n>` controls how many sandboxes run at once and defaults to
+  five, which fits Hobby project limits without changing the boot count.
+- `--blocks <n>` repeats the full matrix inside each boot. More blocks improve
+  a boot's estimate but do not increase the independent sample count.
+- `--runs <n>` forwards the run count to benchmarks that support it.
+- `--vcpus <n>` defaults to 8. Keep it identical across compared cases.
+- `--no-cache` forces a fresh setup snapshot after debugging setup changes.
+- `--keep` retains measurement VMs after collection; omit it normally.
+
+Run the launcher as a background task for long matrices and return to it on
+completion. Relay the printed run directory immediately. Do not stop a run
+because partial output appears favorable; only `summary.md` after all planned
+VMs finish is reportable.
+
+## Read and report results
+
+The launcher writes:
+
+- `meta.json`: source fingerprint, snapshot, cases, allocation, and per-VM
+  status;
+- `summary.md`: absolute boot-level means and paired comparisons;
+- `vm-*/bench-output/`: raw JSON and per-case logs; and
+- `vm-*/bench.log`: the full VM transcript.
+
+Rebuild a summary without rerunning:
+
+```sh
+node .agents/skills/sandbox-bench/scripts/summarize.mjs <run-directory>
+```
+
+Read [references/methodology.md](references/methodology.md) before interpreting
+or reporting a run. In particular:
+
+- Treat the VM boot—not internal iterations or requests—as `n`.
+- Positive deltas mean only "candidate numeric value is larger"; interpret
+  whether that is better from the metric.
+- Require boot-level `p < 0.01`, a 95% CI excluding zero, an A/A-validated
+  team/project, and an independent confirmation before a decision-driving
+  claim.
+- Report every unmatched metric as "no detected difference," not a small win
+  or loss.
+- State VM count, vCPUs, Node/Chrome versions, source fingerprint, exact
+  resolved framework versions, and that percentages are platform-specific.
+
+## Recover or diagnose failures
+
+Start with:
+
+```sh
+node .agents/skills/sandbox-bench/scripts/status.mjs
+```
+
+The runner detaches benchmark work inside each VM, polls it, downloads results,
+and removes VMs after collection. If the local launcher is interrupted, the
+remote work can continue until the configured sandbox timeout. Reconnect and
+collect every completed VM recorded in a run with:
+
+```sh
+node .agents/skills/sandbox-bench/scripts/collect.mjs <run-directory>
+```
+
+The collector leaves still-running VMs alone, updates `meta.json`, removes
+collected VMs, and regenerates `summary.md` once the matrix is complete. Use the
+scoped `vercel sandbox list`, `exec`, and `cp` commands for deeper inspection;
+remove only exact `ftrack-*` sandbox names after recovery.
+
+For a setup or benchmark failure, inspect the relevant per-case log and report
+the failure instead of producing a comparison from an incomplete matrix.

.agents/skills/sandbox-bench/references/methodology.md

@@ -0,0 +1,91 @@
+# Framework-tracker sandbox benchmark methodology
+
+## Use the VM boot as the independent sample
+
+JIT state, heap layout, CPU placement, and host conditions are shared by every
+measurement inside one sandbox boot. Repetitions within a boot improve that
+boot's estimate, but they are not independent evidence.
+
+The sandbox runner therefore:
+
+- prepares one content-addressed source snapshot so every VM starts from the
+  same files, Node version, pnpm version, root dependencies, and (when needed)
+  Chrome installation;
+- runs every matrix case inside every VM;
+- reverses case order across VM/block parity to reduce linear order effects;
+- averages blocks within a VM; and
+- computes confidence intervals and paired deltas across VM boots.
+
+Do not report the framework-tracker CLI's internal browser runs or load-test
+requests as independent samples. The number of completed VMs is `n`.
+
+## Choose comparable cases
+
+- Prefer exact app packages when comparing runtime behavior, for example
+  `app-next-js@canary` and `app-tanstack-start-react`.
+- Use framework selectors when starter and app results are both relevant.
+- The runner maps `@stable` to npm's `latest` tag. Record the resolved
+  `frameworkVersion` from each result rather than assuming a dist-tag stayed
+  fixed.
+- Keep measurements, `--runs`, VM size, source fingerprint, Chrome version,
+  and sandbox project identical across cases.
+- Do not compare results from different run directories as though they were
+  paired. Host and platform changes can dominate small framework effects.
+
+## Allocation and claims
+
+Use one VM only for setup smoke tests. Use at least eight boots for exploratory
+comparisons and sixteen when a small result may drive a decision. More internal
+`--runs` cannot replace more independent boots.
+
+Before making claims for a new Vercel team/project or after a platform,
+benchmark-app, Node, Chrome, or runner change, run an A/A calibration by
+specifying the same target twice. No metric should reach boot-level `p < 0.01`
+beyond the nominal false-positive rate.
+
+Treat a result as detected only when all of these hold:
+
+1. The source fingerprint, snapshot, case arguments, and completed VM count are
+   correct.
+2. The boot-level paired comparison has `p < 0.01`.
+3. The 95% confidence interval excludes zero.
+4. Per-boot signs are not dominated by one outlier.
+5. A decision-driving result repeats in an independent run.
+
+Everything else is "no detected difference," not a small win or loss.
+
+## Metric interpretation
+
+The generated summary uses a mechanical sign convention: a positive delta
+means the candidate produced a larger number. Interpret direction per metric:
+
+- Higher is generally better for `opsPerSec`, `peakRequestsPerSec`, and similar
+  throughput metrics.
+- Lower is generally better for latency, paint timing, INP, build/install time,
+  output size, and error counts.
+- Body size, sample count, total requests, peak workers, and framework version
+  metadata describe the run; they are not automatically performance wins.
+- Browser FP/FCP/INP values are noisy. Check the Chrome version and require
+  stronger replication than for deterministic build size.
+- `ssrLoadTests.stages` is intentionally omitted from the compact summary.
+  Inspect the raw per-VM `ci-stats.json` files when load-curve shape matters.
+
+## Reporting template
+
+Lead with what ran and the independent sample count:
+
+```markdown
+## Next.js canary vs TanStack Start — Vercel Sandbox, 8 boots
+
+Source fingerprint: `<hash>`; Node 24; 8 vCPUs; measurements:
+`ssrLoad`, `ssrRequestThroughput`.
+
+| metric             | candidate delta | 95% CI |      p | verdict  |
+| ------------------ | --------------: | -----: | -----: | -------- |
+| app SSR throughput |          +12.4% |  ±3.1% | 0.0008 | detected |
+
+No detected difference: ...
+```
+
+State that magnitudes are platform-specific. Link or provide the run directory
+and retain `meta.json`, `summary.md`, per-case logs, and raw result JSON files.

.agents/skills/sandbox-bench/scripts/collect.mjs

@@ -0,0 +1,121 @@
+#!/usr/bin/env node
+
+import { execFile } from 'node:child_process'
+import fs from 'node:fs'
+import path from 'node:path'
+import { promisify } from 'node:util'
+import { loadConfig, sandboxScope } from './config.mjs'
+
+const execFileP = promisify(execFile)
+
+function usage() {
+  console.error('Usage: node collect.mjs <run-directory> [--keep]')
+}
+
+async function main() {
+  const runDir = process.argv[2]
+  const keep = process.argv.includes('--keep')
+  if (!runDir || runDir.startsWith('-')) {
+    usage()
+    process.exit(1)
+  }
+
+  const metaPath = path.join(runDir, 'meta.json')
+  const meta = JSON.parse(fs.readFileSync(metaPath, 'utf8'))
+  const config = loadConfig()
+  const scope = sandboxScope(config)
+  const sandbox = async (args) => {
+    const scoped = ['sandbox', ...args]
+    const separator = scoped.indexOf('--')
+    scoped.splice(separator < 0 ? scoped.length : separator, 0, ...scope)
+    const { stdout, stderr } = await execFileP(config.vercelBin, scoped, {
+      maxBuffer: 64 * 1024 * 1024,
+    })
+    return `${stdout}\n${stderr}`
+  }
+  const save = () =>
+    fs.writeFileSync(metaPath, `${JSON.stringify(meta, null, 2)}\n`)
+  let pending = 0
+  let failures = 0
+
+  for (const [index, state] of Object.entries(meta.vms)) {
+    if (state.phase === 'done') continue
+    let done
+    try {
+      const output = await sandbox([
+        'exec',
+        state.vm,
+        '--timeout',
+        '2m',
+        '--',
+        'bash',
+        '-c',
+        'cat /vercel/sandbox/bench.done 2>/dev/null || echo no',
+      ])
+      done = output.match(/(?:^|\n)(no|\d+)(?:\n|$)/)?.[1]
+    } catch (error) {
+      console.error(`${state.vm}: could not read status: ${error.message}`)
+      failures += 1
+      continue
+    }
+    if (done === 'no' || done === undefined) {
+      console.log(`${state.vm}: still running`)
+      pending += 1
+      continue
+    }
+
+    const archive = path.join(runDir, `vm-${index}.tgz`)
+    const directory = path.join(runDir, `vm-${index}`)
+    try {
+      await sandbox([
+        'cp',
+        `${state.vm}:/vercel/sandbox/bench-results.tgz`,
+        archive,
+      ])
+      fs.mkdirSync(directory, { recursive: true })
+      await execFileP('tar', ['-xzf', archive, '-C', directory])
+      const exitCode = Number(
+        fs.readFileSync(path.join(directory, 'bench.exit'), 'utf8').trim(),
+      )
+      state.phase = exitCode === 0 ? 'done' : 'failed'
+      state.exitCode = exitCode
+      state.updatedAt = new Date().toISOString()
+      save()
+      console.log(`${state.vm}: collected (exit ${exitCode})`)
+      if (exitCode !== 0) failures += 1
+      if (!keep) {
+        await sandbox(['rm', state.vm])
+      }
+    } catch (error) {
+      console.error(`${state.vm}: collection failed: ${error.message}`)
+      failures += 1
+    }
+  }
+
+  const states = Object.values(meta.vms)
+  if (
+    pending === 0 &&
+    failures === 0 &&
+    states.length > 0 &&
+    states.every((state) => state.phase === 'done')
+  ) {
+    meta.phase = 'complete'
+    save()
+    const summary = await execFileP(
+      process.execPath,
+      [new URL('./summarize.mjs', import.meta.url).pathname, runDir],
+      { maxBuffer: 64 * 1024 * 1024 },
+    )
+    process.stdout.write(summary.stdout)
+    process.stderr.write(summary.stderr)
+  } else if (failures > 0) {
+    meta.phase = 'failed'
+    save()
+    process.exitCode = 1
+  }
+}
+
+main().catch((error) => {
+  console.error(error instanceof Error ? error.message : error)
+  process.exitCode = 1
+})