Skip to content

chore(evals): converge harness SDK pins onto the workspace catalog - #2743

Open
miguelg719 wants to merge 1 commit into
mainfrom
evals/harness-sdk-catalog
Open

chore(evals): converge harness SDK pins onto the workspace catalog#2743
miguelg719 wants to merge 1 commit into
mainfrom
evals/harness-sdk-catalog

Conversation

@miguelg719

@miguelg719 miguelg719 commented Aug 16, 2026

Copy link
Copy Markdown
Collaborator

What

Moves both external-harness SDK deps in packages/evals onto the workspace catalog:

  • @anthropic-ai/claude-agent-sdk: ^0.2.141catalog: (0.3.224)
  • @openai/codex-sdk: 0.125.0catalog: (0.147.0)

Why

#2719 and #2720 added both SDKs to the catalog for the integration examples, forking the versions evals pinned directly — two copies of each SDK in the monorepo. This converges on one version per SDK, as a standalone pre-step to the broader harness-adapter consolidation.

Compat verification

Evals types both SDKs structurally (dynamic import), so tsc cannot catch drift — verified against the installed .d.ts instead:

  • claude-agent-sdk 0.3.224: every query() option evals passes exists — canUseTool (updatedInput now optional), systemPrompt preset+append, settingSources, mcpServers, pathToClaudeCodeExecutable, createSdkMcpServer/tool. Publish history shows 0.3.142 directly succeeds 0.2.141 (version-scheme continuation).
  • codex-sdk 0.147.0: startThread options (sandboxMode, approvalPolicy, networkAccessEnabled, webSearchMode, skipGitRepoCheck), runStreamed({outputSchema, signal}), constructor config bag (arbitrary --config overrides, so show_raw_agent_reasoning still flows) all present; ThreadEvent kinds unchanged.

Testing

  • turbo run build typecheck --filter @browserbasehq/stagehand-evals
  • Evals unit suite: 420/420
  • Connected smoke (evals run b:webvoyager --harness claude_code|codex --tool stagehand_code -l 1 -e browserbase): pending — blocked on Browserbase credentials in the runner env; both attempts failed at Stagehand init before any SDK code executed. Draft until the smoke passes.

Summary by cubic

Converges packages/evals harness SDK dependencies to the workspace catalog to remove duplicate SDK pins and align with the integration examples. Old behavior: evals pinned @anthropic-ai/claude-agent-sdk ^0.2.141 and @openai/codex-sdk 0.125.0; new behavior: both use catalog: (0.3.224 and 0.147.0). Side effect: lockfile updates only; no runtime changes expected.

  • Verified against installed types: claude-agent-sdk@0.3.224 retains all query() options used by evals; codex-sdk@0.147.0 retains startThread, runStreamed, and constructor config behaviors with unchanged ThreadEvent kinds.
  • Evals unit tests pass (420/420). Connected smoke pending due to missing Browserbase credentials; failures occurred before any SDK calls.

Written for commit ae1ccc8. Summary will update on new commits.

Review in cubic

PRs #2719/#2720 added @anthropic-ai/claude-agent-sdk (0.3.224) and
@openai/codex-sdk (0.147.0) to the pnpm catalog for the integration
examples, forking the versions evals pinned directly (^0.2.141 and
0.125.0). Move both evals deps to catalog: so the monorepo tracks a
single version per SDK.

Compat verified against the installed types: every query() option the
claude_code harness passes (canUseTool, systemPrompt preset+append,
settingSources, mcpServers, createSdkMcpServer/tool) exists in 0.3.224,
and every codex option (startThread config, runStreamed outputSchema/
signal, constructor config bag incl. show_raw_agent_reasoning) exists
in 0.147.0 with unchanged ThreadEvent kinds. Evals unit suite passes
(420 tests).
@changeset-bot

changeset-bot Bot commented Aug 16, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: ae1ccc8

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@miguelg719

Copy link
Copy Markdown
Collaborator Author

Connected smokes now verified with real credentials:

  • claude_code (b:webvoyager --tool stagehand_code -l 1 -e browserbase): 3/3 trials passed on claude-agent-sdk 0.3.224 (run also exercised the feat(integrations): add shared harness contract module to core #2746 re-export path — evidence covers both).
  • codex (same suite): harness ran end-to-end on codex-sdk 0.147.0 — threads streamed, 50-step budget enforced, trajectories graded; 1/3 trials passed (failures were task-level step-budget exhaustion, not SDK plumbing).

TypeScript unit CI failure is unrelated and repo-wide: packages/docs sdk-reference suite exceeds vitest's 5s default on CI runners since #2698 grew the MDX scan set — fix in #2747. Will rerun checks once that merges.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 2 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.
Architecture diagram
sequenceDiagram
    participant EV as @browserbasehq/stagehand-evals
    participant WS as Workspace Catalog
    participant CL as claude-agent-sdk
    participant CX as codex-sdk
    participant PKG as Package Manager (turbo/pnpm)

    Note over EV,PKG: Dependency Resolution Flow

    EV->>WS: Declare @anthropic-ai/claude-agent-sdk: "catalog:"
    EV->>WS: Declare @openai/codex-sdk: "catalog:"
    WS-->>PKG: Resolve to claude-agent-sdk 0.3.224
    WS-->>PKG: Resolve to codex-sdk 0.147.0
    PKG->>CL: Pin to 0.3.224 (shared with integration examples)
    PKG->>CX: Pin to 0.147.0 (shared with integration examples)

    Note over EV,PKG: Build & Type Verification

    EV->>PKG: turbo build (evals)
    PKG->>CL: Compile-time imports
    PKG->>CX: Compile-time imports
    CL-->>PKG: Installed .d.ts (0.3.224)
    CX-->>PKG: Installed .d.ts (0.147.0)
    PKG->>PKG: Verify structural types (dynamic import)

    Note over EV: Runtime harness adapter flow (unchanged behavior)

    EV->>CL: query() with options (canUseTool, systemPrompt, mcpServers, pathToClaudeCodeExecutable)
    CL-->>EV: Streamed events (unchanged)

    EV->>CX: startThread({sandboxMode, approvalPolicy, networkAccessEnabled, webSearchMode, skipGitRepoCheck})
    CX-->>EV: ThreadEvent kinds (unchanged)

    Note over EV: Downstream dependency graph

    CL->>PKG: Type-level import (constructor config, createSdkMcpServer)
    CX->>PKG: Type-level import (runStreamed with outputSchema, signal)
Loading

Re-trigger cubic

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant