fix(integrations): harden fx facade discovery and screenshots

[miguelg719]

why

The fx harness added in #2776 can miss all three Stagehand tools during dynamic search and then invent legacy navigate tool names. Inline full-page PNG responses can also exceed the fx raw MCP frame cap before tool-result truncation runs, which closes the RPC connection.

what changed

• add deterministic fx project guidance with the exact three tool names and explicit page.goto navigation
• make the facade run and screenshot descriptions easier to discover and harder to misinterpret
• add an fx-specific 60 KB screenshot payload budget that defaults to viewport JPEG, retries progressively smaller JPEGs, and returns a small tool error if no image fits
• document the distinction between the raw response-frame cap and max_tool_result_bytes
• add focused contract, transport-budget, and fx configuration tests

test plan

• pnpm --filter @browserbasehq/stagehand-extension build
• pnpm --filter @browserbasehq/stagehand build
• pnpm --filter @browserbasehq/stagehand-integrations typecheck
• pnpm --filter @browserbasehq/stagehand-integrations test
• pnpm exec oxlint on all changed TypeScript files
• pnpm exec oxfmt --check on all changed files
• git diff --check

---

Summary by cubic

Hardens fx Stagehand facade tool discovery and screenshot transport to prevent stalled runs and dropped MCP connections, and publishes a complete fx integration guide. Previously fx could return no tools and models invented legacy navigate names; full‑page PNG screenshots could exceed fx’s raw frame cap and close the connection. Now discovery is deterministic and screenshots respect a configurable base64 budget with safe, clamped JPEG retries.

• Deterministic discovery and navigation: pins exactly three tools (“run”, “snapshot”, “screenshot”) with updated descriptions; there is no separate navigate/start tool. packages/integrations/fx/AGENTS.md, the skills/stagehand-facade skill, and new docs (/v4/integrations/fx) instruct agents to call mcp_stagehand_run, mcp_stagehand_snapshot, and mcp_stagehand_screenshot directly and navigate with await page.goto(...).
• Transport-safe screenshots: the facade enforces a base64 budget (--max-screenshot-base64-bytes=...). The fx template sets --max-screenshot-base64-bytes=60000, defaults unspecified screenshots to a viewport JPEG (quality 40), retries with smaller JPEG qualities (40 → 25 → 10) without ever increasing a requested JPEG quality, and returns a small tool error if none fit. Docs clarify the raw response-frame cap vs max_tool_result_bytes.
• Tests: add contract, transport-budget, and fx configuration tests, including quality clamping during screenshot retries.
• Required for custom fx setups: add --max-screenshot-base64-bytes=60000 (or a suitable value) to the Stagehand MCP command; run fx from packages/integrations/fx so it loads the project guidance; prefer {"type":"jpeg","quality":40,"fullPage":false} for screenshots.

^{Written for commit 9c5d64a. Summary will update on new commits.}

[changeset-bot]

⚠️ No Changeset found

Latest commit: 9c5d64a

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[cubic-dev-ai]

All reported issues were addressed across 10 files

Architecture diagram

sequenceDiagram
    participant Agent as AI Agent (fx)
    participant Skill as Skill/AGENTS Docs
    participant MCP as MCP Client (fx)
    participant Server as MCP Server
    participant Tool as Tool Registry
    participant Facade as Stagehand Facade
    participant Budget as Screenshot Budget Guard
    participant Browser as Playwright Browser

    Note over Agent,Browser: fx discovery flow - PR makes tool names deterministic
    Agent->>Skill: Read AGENTS.md + stagehand-facade skill
    Skill-->>Agent: Exact 3 tool names + usage patterns
    Agent->>MCP: Select tools: mcp_stagehand_run/snapshot/screenshot
    MCP->>Server: Initialize MCP stdio connection
    Server->>Tool: Register 3 tools (run, snapshot, screenshot)
    Tool-->>Agent: Confirm exact tool names

    Note over Agent,Server: Snapshot flow (unchanged)
    Agent->>MCP: call_tool(stagehand_snapshot)
    MCP->>Server: CallToolRequest snapshot
    Server->>Tool: tools.snapshot(request)
    Tool-->>Server: Page snapshot with element IDs
    Server-->>Agent: Snapshot result

    Note over Agent,Server: Run / navigate flow - PR clarifies no separate navigate tool
    Agent->>MCP: call_tool(stagehand_run) with page.goto()
    MCP->>Server: CallToolRequest run
    Server->>Tool: tools.run(request)
    Tool->>Browser: Execute JavaScript / actions
    Browser-->>Tool: Execution result
    Tool-->>Server: Result
    Server-->>Agent: Run result

    Note over Agent,Server: Screenshot with optional budget guard
    Agent->>MCP: call_tool(stagehand_screenshot)
    MCP->>Server: CallToolRequest screenshot

    alt Budget NOT configured (no CLI flag)
        Server->>Tool: tools.screenshot(request)
        Tool->>Browser: Capture screenshot
        Browser-->>Tool: Image data
        Tool-->>Server: Image data
        Server-->>Agent: Screenshot (no compression)
    else Budget configured (--max-screenshot-base64-bytes)
        Server->>Budget: captureScreenshotWithinBase64Budget(request, budget)
        Note over Budget: Uses requested options first
        Budget->>Tool: Capture with requested options
        Tool->>Browser: Screenshot capture (original opts)
        Browser-->>Tool: Image data
        Tool-->>Budget: Base64 result
        alt Size exceeds budget
            Budget->>Budget: Retry with viewport JPEG q40 → q25 → q10
            Budget->>Tool: Capture retry (progressive compression)
            Tool->>Browser: Screenshot (smaller JPEG)
            Browser-->>Tool: Compressed image
            Tool-->>Budget: Image data
            Note over Budget: If still exceeds budget after all retries
            Budget-->>Server: Throw error (small tool failure)
        else Image fits budget
            Budget-->>Server: Captured image
        end
    end
    Server-->>Agent: Screenshot (image or error)

    Note over MCP,Server: Response frame safety<br/>Raw screenshots can exceed fx MCP frame cap<br/>(separate from max_tool_result_bytes)

Loading

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>