Blog: benchmark transparency update for the AI agents post

[AvdLee]

Summary

Transparency update to "How We Test AI Agents for the iOS Simulator", following Cameron Cooke's public feedback that the post claimed "118.88x fewer output bytes compared with other tools" without naming them.

• Names the compared tools with pinned versions everywhere: AXe 1.8.0 and XcodeBuildMCP 2.7.0 (driven via its stdio MCP server interface).
• Adds a disclaimer in Antoine's words: AXe is deliberately a raw, non-agentic primitive layer by design; XcodeBuildMCP builds on top of it with state management and compact output. AXe's large output is framed as a design choice, not a flaw.
• Attributes the old large multipliers (118.88x / 21.06x) to the raw-AXe comparison and adds the new XcodeBuildMCP head-to-head numbers from the frozen 2026-07-31 three-round run: 3.0x tool-response bytes, 8.2x byte-based context estimate, 171.5s slower over the 14 common scenarios.
• Leads the findings with correctness and capability coverage instead of the byte multiplier, including RocketSim's own 7 wrong interactions and 1 unexpected failure (Face ID, round 3).
• Adds methodology notes: SHA-256-pinned tools, three rounds, byte-based context proxy definition (MCP tool catalog charged symmetrically with skill files; a reproducible proxy, not billed tokens), head-to-head vs capability-only separation, and frozen scripts that keep iOS 26 Settings drift failures visible.
• Updates record counts to the 2026-07-31 run (156 records: 150 head-to-head, 6 capability-only) and bumps modifiedTime.

All numbers come from the frozen benchmark report in AvdLee/RocketSim#1336 (docs/agent-protocol/eval/runs/2026-07-31-agent-skill-benchmark/).

Merge order

Do NOT merge before https://github.com/AvdLee/RocketSim/pull/1336 is merged, so the published post always points at a benchmark run that exists on the RocketSim main branch. Antoine also wants to share the numbers with Cameron Cooke before this goes live.

Validation

From docs/: npm run lint, npm run format:check, npm run typecheck, npm run build, and npm run knip all pass (exit 0).