feat: run heuristic discovery through Codex app-server - #78
Open
wunitb wants to merge 1 commit into
Open
Conversation
Studio's "Start heuristic scan" (DISCOVERY_SCOUT) ran the bounded discovery tool loop in-process against chatgpt.com/backend-api/codex/responses with a third-party originator. The ChatGPT-subscription OAuth cache is rejected on that route (HTTP 403 on the zero-inference usage probe), so the scan stayed permanently blocked while every other Agent workload already passed preflight through the local `codex app-server`. - Add `CodexAppServerAgentPort`: the same `DiscoveryAgentSession` tool loop, driven through app-server dynamic tools (thread/start, turn/start, item/tool/call -> respond, token-usage step boundary, turn/completed, fail-closed built-in effects, interrupt + close on every exit). A step is a model response that carried a tool call or ended the turn; commentary-only responses count as provider requests but not against the step budget. - Extract the shared loop contract from `discovery-agent.ts` (`DISCOVERY_AGENT_TOOL_MANIFEST`, instructions, prompt, legal tool surface, tool execution, stop predicate and termination ladder) so both transports advertise one identical tool contract and label runs the same way. Calls outside the legal surface are retained as rejected session effects that consume the bounded budget without poisoning idempotent replay. - `createCodexDiscoveryRuntime` gains `transport` (env `PMH_CODEX_DISCOVERY_TRANSPORT`, default `app-server`) and reports `transport: CODEX_APP_SERVER`; the direct Responses route stays an explicit opt-in. - The legacy DISCOVERY_SCOUT route binds the shared Codex app-server runtime definition under its own profile/route keys (the store is unique on key+revision); every legacy import states its transport, the server derives it from the live model runtime, and route selection prefers the newest-revision route whose runtime matches that transport. - The app-server connection factory tears down the spawned child when the initialize handshake fails instead of orphaning it. Verified on a fresh operator desk: preflight `CODEX_APP_SERVER_ACCOUNT -> USABLE / ELIGIBLE`, two operator-triggered scans completed `PASS` through a `codex app-server --stdio` child (one retained falsification, one MODEL_FINISHED); workspace type-check and the full control-plane suite pass.
Author
|
Final certification on the committed code: operator-triggered scan from the Studio button completed PASS in 111 s through a |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Studio's Start heuristic scan (task kind
DISCOVERY_SCOUT) ran the bounded discovery tool loop in-process through the Vercel AI SDK againsthttps://chatgpt.com/backend-api/codex/responseswithoriginator: prediction-market-harness. The ChatGPT-subscription OAuth cache is rejected on that route: the zero-inferencecodex/usageprobe returns HTTP 403, so the route's execution profile staysAUTH_REJECTED / BLOCKEDand the scan button is permanently disabled. Every other Agent workload already runs through the localcodex app-server(the first-party client the subscription accepts) and passes its account preflight. PLANS.md already named "an app-server-backed Codex runtime" as the next usable-route candidate for this legacy workload.Change
CodexAppServerAgentPort(packages/control-plane/src/codex-app-server-model.ts): aDiscoveryAgentPortthat drives the sameDiscoveryAgentSessiontool loop through app-server dynamic tools —thread/start(read-only, ephemeral, six dynamic tools),turn/start,item/tool/call→ respond,thread/tokenUsage/updatedas the step boundary,turn/completed, retrying/terminalerrornotifications, fail-closed built-in item types,turn/interrupt+close+ temp-dir removal on every exit. A step is a model response that carried a tool call or ended the turn; commentary-only responses count as provider requests but not against the step budget.discovery-agent.ts:DISCOVERY_AGENT_TOOL_MANIFEST,discoveryAgentInstructions,discoveryAgentPrompt,activeDiscoveryToolNames,executeDiscoveryTool,discoveryLoopShouldStop,discoveryTerminationReason. The AI SDK loop consumes them; its tool descriptions, schemas, instructions, prompt JSON,prepareStepgating and termination ladder are unchanged (verified line by line against the diff). Calls outside the legal surface are retained by the session asREJECTED / PROTOCOL_INVALIDeffects that consume the bounded budget without registering the idempotent-replay key.createCodexDiscoveryRuntimegainstransport(envPMH_CODEX_DISCOVERY_TRANSPORT, defaultapp-server) and reportsmodelProvider.transport = CODEX_APP_SERVER; the direct Responses route stays an explicitresponsesopt-in.importLegacyAiRuntimeConfiguration(configuration, { discoveryTransport })is now explicit at every call site. WithAPP_SERVERtheDISCOVERY_SCOUTroute binds the sharedCODEX / codex-app-server-v2:0.147runtime definition under its own profile/route keys (legacy-discovery-app-server*; the SQLite store is unique on key + revision, so both compositions can be retained for one configuration revision). The server derives the transport from the live model runtime for both import sites and prefers, among the newest-revisionDISCOVERY_SCOUTroutes, the one whose runtime matches that transport.initializehandshake now closes the spawnedcodex app-serverchild instead of orphaning it..env.example,docs/OPERATIONS.md,PLANS.md.Authority boundary unchanged: the thread is read-only/ephemeral, built-in effects fail the run closed, the session remains the only writer of tool effects, and no projection, log or test carries the OAuth token.
Evidence
Fresh operator desk (macOS, Node 22.23, codex-cli 0.149.0, ChatGPT Pro OAuth):
POST /execution-profiles/<discovery>/preflight→CODEX_USAGE / AUTH_REJECTED / BLOCKED(HTTP 403); the eight app-server routes →USABLE / ELIGIBLE.runtime.kind = CODEX, preflight →CODEX_APP_SERVER_ACCOUNT / USABLE / ELIGIBLE, and the Studio badge readsCODEX · gpt-5.6-terra · USABLE.PASSthrough acodex app-server --stdiochild of the control plane: one retained falsification (stepCount 8 → STEP_LIMITunder the pre-fix accounting), oneMODEL_FINISHEDin 49 s (stepCount 2, toolCallCount 1), and one after the review fixes (see the final comment). No app-server child remained after completion.pnpm -r checkpasses; control-plane suite 121 files / 819 tests pass. New coverage: 14 port cases (dynamic-tool loop, out-of-surface and unknown tools, batched calls, abandoned pending call, budget labels, unclassified/timeout/deadline failures, retrying and terminal error notifications, protocol violations, built-in effects, model mismatch, transport default and opt-in), the legacy app-server composition, and the child teardown on a failed handshake.PASS_WITH_FINDINGS; all MAJOR and MINOR findings are addressed in this branch (child-process leak on handshake failure, shared stop/termination predicates, session-recorded out-of-surface calls, single source for the discovery transport,mkdtempinside the failure boundary, abandoned-call handling onturn/completed, explicit transport at every legacy import).Rollback / opt-out
PMH_CODEX_DISCOVERY_TRANSPORT=responsesrestores the previous in-process Responses transport and itsHARNESS_IN_PROCESSroute without a code change. Retained routes are immutable; the previouslegacy-discovery-defaultroute stays in the store and is selected again under that transport.Not in this PR
accessDriverof the reused Codex model profile still saysCODEX_RESPONSES(matches the existing portfolio convention for the rule-evidence app-server profiles); introducing a distinctCODEX_APP_SERVERdriver is a follow-up.mediumthrough Studio because two Terra/high mechanism-research runs stalled silently past the 300 s run budget; that is a durable per-desk setting, not part of this change.