Skip to content

feat: run heuristic discovery through Codex app-server - #78

Open
wunitb wants to merge 1 commit into
TraderAlice:mainfrom
wunitb:codex/discovery-scout-app-server
Open

feat: run heuristic discovery through Codex app-server#78
wunitb wants to merge 1 commit into
TraderAlice:mainfrom
wunitb:codex/discovery-scout-app-server

Conversation

@wunitb

@wunitb wunitb commented Aug 24, 2026

Copy link
Copy Markdown

Problem

Studio's Start heuristic scan (task kind DISCOVERY_SCOUT) ran the bounded discovery tool loop in-process through the Vercel AI SDK against https://chatgpt.com/backend-api/codex/responses with originator: prediction-market-harness. The ChatGPT-subscription OAuth cache is rejected on that route: the zero-inference codex/usage probe returns HTTP 403, so the route's execution profile stays AUTH_REJECTED / BLOCKED and the scan button is permanently disabled. Every other Agent workload already runs through the local codex app-server (the first-party client the subscription accepts) and passes its account preflight. PLANS.md already named "an app-server-backed Codex runtime" as the next usable-route candidate for this legacy workload.

Change

  • CodexAppServerAgentPort (packages/control-plane/src/codex-app-server-model.ts): a DiscoveryAgentPort that drives the same DiscoveryAgentSession tool loop through app-server dynamic tools — thread/start (read-only, ephemeral, six dynamic tools), turn/start, item/tool/call → respond, thread/tokenUsage/updated as the step boundary, turn/completed, retrying/terminal error notifications, fail-closed built-in item types, turn/interrupt + close + temp-dir removal on every exit. A step is a model response that carried a tool call or ended the turn; commentary-only responses count as provider requests but not against the step budget.
  • Shared loop contract extracted from discovery-agent.ts: DISCOVERY_AGENT_TOOL_MANIFEST, discoveryAgentInstructions, discoveryAgentPrompt, activeDiscoveryToolNames, executeDiscoveryTool, discoveryLoopShouldStop, discoveryTerminationReason. The AI SDK loop consumes them; its tool descriptions, schemas, instructions, prompt JSON, prepareStep gating and termination ladder are unchanged (verified line by line against the diff). Calls outside the legal surface are retained by the session as REJECTED / PROTOCOL_INVALID effects that consume the bounded budget without registering the idempotent-replay key.
  • Transport selection: createCodexDiscoveryRuntime gains transport (env PMH_CODEX_DISCOVERY_TRANSPORT, default app-server) and reports modelProvider.transport = CODEX_APP_SERVER; the direct Responses route stays an explicit responses opt-in.
  • Substrate: importLegacyAiRuntimeConfiguration(configuration, { discoveryTransport }) is now explicit at every call site. With APP_SERVER the DISCOVERY_SCOUT route binds the shared CODEX / codex-app-server-v2:0.147 runtime definition under its own profile/route keys (legacy-discovery-app-server*; the SQLite store is unique on key + revision, so both compositions can be retained for one configuration revision). The server derives the transport from the live model runtime for both import sites and prefers, among the newest-revision DISCOVERY_SCOUT routes, the one whose runtime matches that transport.
  • Transport factory: a failed initialize handshake now closes the spawned codex app-server child instead of orphaning it.
  • Docs: .env.example, docs/OPERATIONS.md, PLANS.md.

Authority boundary unchanged: the thread is read-only/ephemeral, built-in effects fail the run closed, the session remains the only writer of tool effects, and no projection, log or test carries the OAuth token.

Evidence

Fresh operator desk (macOS, Node 22.23, codex-cli 0.149.0, ChatGPT Pro OAuth):

  • Before: POST /execution-profiles/<discovery>/preflightCODEX_USAGE / AUTH_REJECTED / BLOCKED (HTTP 403); the eight app-server routes → USABLE / ELIGIBLE.
  • After: the discovery route reports runtime.kind = CODEX, preflight → CODEX_APP_SERVER_ACCOUNT / USABLE / ELIGIBLE, and the Studio badge reads CODEX · gpt-5.6-terra · USABLE.
  • Three operator-triggered scans from the Studio button completed PASS through a codex app-server --stdio child of the control plane: one retained falsification (stepCount 8 → STEP_LIMIT under the pre-fix accounting), one MODEL_FINISHED in 49 s (stepCount 2, toolCallCount 1), and one after the review fixes (see the final comment). No app-server child remained after completion.
  • pnpm -r check passes; control-plane suite 121 files / 819 tests pass. New coverage: 14 port cases (dynamic-tool loop, out-of-surface and unknown tools, batched calls, abandoned pending call, budget labels, unclassified/timeout/deadline failures, retrying and terminal error notifications, protocol violations, built-in effects, model mismatch, transport default and opt-in), the legacy app-server composition, and the child teardown on a failed handshake.
  • Review: a second independent model review (Anthropic-family, not the OpenAI codex client) returned PASS_WITH_FINDINGS; all MAJOR and MINOR findings are addressed in this branch (child-process leak on handshake failure, shared stop/termination predicates, session-recorded out-of-surface calls, single source for the discovery transport, mkdtemp inside the failure boundary, abandoned-call handling on turn/completed, explicit transport at every legacy import).

Rollback / opt-out

PMH_CODEX_DISCOVERY_TRANSPORT=responses restores the previous in-process Responses transport and its HARNESS_IN_PROCESS route without a code change. Retained routes are immutable; the previous legacy-discovery-default route stays in the store and is selected again under that transport.

Not in this PR

  • The accessDriver of the reused Codex model profile still says CODEX_RESPONSES (matches the existing portfolio convention for the rule-evidence app-server profiles); introducing a distinct CODEX_APP_SERVER driver is a follow-up.
  • The operator desk used for this evidence had its reasoning effort switched to medium through Studio because two Terra/high mechanism-research runs stalled silently past the 300 s run budget; that is a durable per-desk setting, not part of this change.

Studio's "Start heuristic scan" (DISCOVERY_SCOUT) ran the bounded discovery
tool loop in-process against chatgpt.com/backend-api/codex/responses with a
third-party originator. The ChatGPT-subscription OAuth cache is rejected on
that route (HTTP 403 on the zero-inference usage probe), so the scan stayed
permanently blocked while every other Agent workload already passed preflight
through the local `codex app-server`.

- Add `CodexAppServerAgentPort`: the same `DiscoveryAgentSession` tool loop,
  driven through app-server dynamic tools (thread/start, turn/start,
  item/tool/call -> respond, token-usage step boundary, turn/completed,
  fail-closed built-in effects, interrupt + close on every exit). A step is a
  model response that carried a tool call or ended the turn; commentary-only
  responses count as provider requests but not against the step budget.
- Extract the shared loop contract from `discovery-agent.ts`
  (`DISCOVERY_AGENT_TOOL_MANIFEST`, instructions, prompt, legal tool surface,
  tool execution, stop predicate and termination ladder) so both transports
  advertise one identical tool contract and label runs the same way. Calls
  outside the legal surface are retained as rejected session effects that
  consume the bounded budget without poisoning idempotent replay.
- `createCodexDiscoveryRuntime` gains `transport` (env
  `PMH_CODEX_DISCOVERY_TRANSPORT`, default `app-server`) and reports
  `transport: CODEX_APP_SERVER`; the direct Responses route stays an explicit
  opt-in.
- The legacy DISCOVERY_SCOUT route binds the shared Codex app-server runtime
  definition under its own profile/route keys (the store is unique on
  key+revision); every legacy import states its transport, the server derives
  it from the live model runtime, and route selection prefers the
  newest-revision route whose runtime matches that transport.
- The app-server connection factory tears down the spawned child when the
  initialize handshake fails instead of orphaning it.

Verified on a fresh operator desk: preflight `CODEX_APP_SERVER_ACCOUNT ->
USABLE / ELIGIBLE`, two operator-triggered scans completed `PASS` through a
`codex app-server --stdio` child (one retained falsification, one
MODEL_FINISHED); workspace type-check and the full control-plane suite pass.
@wunitb

wunitb commented Aug 24, 2026

Copy link
Copy Markdown
Author

Final certification on the committed code: operator-triggered scan from the Studio button completed PASS in 111 s through a codex app-server --stdio child of the control plane (stepCount 3, toolCallCount 3, EXPLICIT_COMPLETION); no app-server child was left running afterwards. Full control-plane suite on this branch: 121 files / 820 tests, pnpm -r check clean.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant