[RNE Rewrite] test: add an on-device performance regression harness

[msluszniak]

Description

Adds apps/benchmarks, a headless Expo app that runs the task pipelines against deterministic synthetic inputs and reports model load time, inference latency and peak memory as JSON, plus a driver that collects a run and a comparator that diffs two runs and fails on regressions.

Built to bracket the ExecuTorch 1.3.1 to 1.4.1 bump: run the suite on 1.3.1, bump, run it again on the same device, compare.

Three design points worth a look:

• A raw-execute pass isolates ExecuTorch from the pipeline. A task timing folds model.execute together with preprocessing and post-processing, which are TypeScript and unaffected by a bump. The pass loads the .pte on its own and sizes its tensors from model.schema, so it covers every method a program exports and needs no per-model wiring. Methods whose schema cannot be pinned to concrete shapes are reported as skipped, with the reason.
• Memory is sampled in a pass of its own. Reading total PSS on Android walks /proc/self/smaps and costs milliseconds, which would otherwise land in the inference numbers. modules/bench-probe is a local Expo module reading task_vm_info.phys_footprint on iOS and total PSS on Android, both of which count the resident pages of a memory mapped .pte that the native heap counters miss.
• Inputs are pure functions of their parameters, so two runs feed byte identical data. Post-processing cost is input dependent, so a harness reading a photo off the device would move for reasons unrelated to the change under test.

The comparator refuses to diff runs from different devices, and reports a metric whose workload changed as INCOMPARABLE rather than as a delta. A delta inside the run's own interquartile range is reported but not failed.

Introduces a breaking change?

• Yes
• No

Type of change

• Bug fix (change which fixes an issue)
• New feature (change which adds functionality)
• Documentation update (improves or adds clarity to existing documentation)
• Other (chores, tests, code style improvements etc.)

Tested on

• iOS
• Android

Testing instructions

cd apps/benchmarks
yarn bench --platform android --label et-1.3.1     # builds, launches, collects
yarn bench --platform android --label et-1.4.1     # after the bump
yarn bench:compare results/et-1.3.1-*.json results/et-1.4.1-*.json

yarn bench starts a collector, sets the app's EXPO_PUBLIC_BENCH_* variables, builds and launches the app, and writes results/<label>-<platform>-<device>.json. It sets up adb reverse on Android and binds to the host LAN address for an iOS device. --suite full adds the larger models, --only <id> runs a single case, --no-launch waits for an app started by hand.

The full quick suite was run on a Galaxy S26 Ultra (SM-S948B, Android 16) with all six cases reporting. A full suite run covering Whisper and Supertonic was in progress at the time of writing.

Screenshots

Related issues

Closes #1078

Checklist

• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas
• I have updated the documentation accordingly
• My changes generate no new warnings

Additional notes

Not yet run on iOS. The CoreML classification case is in the suite but unexercised, since CoreML programs fail to encode on the Simulator and need a physical device.

Two things the device runs surfaced that are worth knowing when reading a report:

• execute.<method> and pipeline.median are not comparable to each other. The raw pass takes dynamic dimensions at the top of their declared domain, so on all-MiniLM-L6-v2 it runs a 254 token forward while the pipeline runs a 20 token one, and the raw number comes out several times larger. Each is comparable against itself across runs, which is all the comparator asks. The resolved shapes are recorded per method.
• Style transfer peaks at roughly 1257 MB against a 372 MB loaded footprint on the S26 Ultra. That is the first measurement rather than a regression against anything, but it may be worth a look independent of the bump.

The suite is triggered by hand rather than run in CI. A host side benchmark on a CI runner would exercise none of the backends a bump actually changes, and its noise floor is wider than most regressions.