diff --git a/CHANGELOG.md b/CHANGELOG.md index 512a0c0b..c824b1d5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,16 @@ ## Unreleased +## 0.125.0 + +### Steering policy is registry data; the loop-kernel steering-driver module is deleted + +BREAKING. `src/runtime/steering-drivers.ts` is deleted, and with it the kernel exports `steeringDriver`, `naiveDriver`, `dumbDriver`, `SteeringDirectiveData`, `SteeringDecision`, `ApplyContinuation`, `NaiveDriverOptions`, and `DumbDriverOptions`. +The steering POLICY texts stay where the graph reads them — registry data (`delegates/naive-continuation`, `delegates/dumb-continuation-pass` / `-fail` in the kernel prompt registry) a `delegates` edge attaches as versioned directives — so the naive/dumb control policies remain optimizable data rows; only the loop-kernel interpreter function is gone. +`defineLeaderboard`'s per-cell retry — the module's one consumer — is now `naiveRetryDriver` in `define-leaderboard.ts` with identical observable behavior: the same `'naive'` trace name, the same plan/decide semantics (re-run the same case verbatim until a shot is `valid` or the shot cap; `pick-winner` on any valid shot, else `refine` under the cap, `fail` at it), and the same leak-free firewall (reads only `verdict.valid`, never `notes`/`scores`). + +Callers to update: anything importing `steeringDriver` / `naiveDriver` / `dumbDriver` from the kernel entry writes the equivalent bare `Driver` literal at its call site (plan: re-issue the task with the continuation folded in until valid or cap; decide: `pick-winner` on any valid shot, else `refine` under the cap, `fail` at it), carrying its continuation texts as its own data or as prompt-registry entries. + ## 0.124.0 ### The analyst can be a tool-equipped agent, and graphs watch their workers diff --git a/docs/api/primitive-catalog.md b/docs/api/primitive-catalog.md index 3a16b688..c3dd18b5 100644 --- a/docs/api/primitive-catalog.md +++ b/docs/api/primitive-catalog.md @@ -7,7 +7,7 @@ # Primitive catalog — the never-stale anti-reinvention inventory -> **GENERATED** from `@tangle-network/agent-runtime@0.124.0` and `@tangle-network/agent-eval@0.142.2` by `scripts/gen-primitive-catalog.mjs`. Do NOT hand-edit — run `pnpm run docs:api`. This is the mechanical companion to the JUDGMENT in `canonical-api.md` (§2 decision table + §1.5 AgentProfile law): that doc says WHICH primitive to reach for and what NOT to build; this catalog proves WHAT exists. Per-symbol signatures + `file:line` live in the per-module pages under `docs/api/`. +> **GENERATED** from `@tangle-network/agent-runtime@0.125.0` and `@tangle-network/agent-eval@0.142.2` by `scripts/gen-primitive-catalog.mjs`. Do NOT hand-edit — run `pnpm run docs:api`. This is the mechanical companion to the JUDGMENT in `canonical-api.md` (§2 decision table + §1.5 AgentProfile law): that doc says WHICH primitive to reach for and what NOT to build; this catalog proves WHAT exists. Per-symbol signatures + `file:line` live in the per-module pages under `docs/api/`. ## 1. agent-runtime — own public surface @@ -510,7 +510,7 @@ Import from `@tangle-network/agent-runtime/intelligence` — 166 exports. ### Execution kernel — recursive atom, supervision, executors, round-synchronous loop -Import from `@tangle-network/agent-runtime/kernel` — 717 exports. +Import from `@tangle-network/agent-runtime/kernel` — 709 exports. | Symbol | Kind | Summary | |---|---|---| @@ -588,7 +588,6 @@ Import from `@tangle-network/agent-runtime/kernel` — 717 exports. | `deterministicCompletion` | function | Completion for a DETERMINISTIC check (build/test/lint/citation/proof): done iff the check | | `discriminatingMeans` | function | Strategy means recomputed over the DISCRIMINATING tasks only — tasks where the field | | `driverAgent` | function | Build the intelligent recursive driver. Its `act` is the LLM tool-loop; spawn it as a | -| `dumbDriver` | function | Thin compatibility wrapper over {@link steeringDriver} for the dumb (pass/fail-only) control. | | `effectiveConcurrency` | function | The ONE honest effective limit on simultaneous workers: the minimum of the caps that actually | | `envKeyProvider` | function | The env-backed provider: reads the (dotenvx-loaded) process env. Empty / | | `equalKOnCost` | function | Assert the arms are comparable at EQUAL conserved COST (tokens + usd), NOT raw iteration | @@ -623,7 +622,6 @@ Import from `@tangle-network/agent-runtime/kernel` — 717 exports. | `materializeLocalMcp` | function | Spawn every explicitly trusted stdio server in `profile.mcp` as a same-host | | `materializeTreeView` | function | Materialize a recorded `TreeView` from a journaled event list for inspection. Folds | | `modelAuthoredChecks` | function | Default authored-check source: one metered LLM call per task, before sampling, | -| `naiveDriver` | function | Thin compatibility wrapper over {@link steeringDriver} for the naive (no-signal) control. | | `noProgressFor` | function | "Nothing new has happened." Fires when the run has produced no new settled work for `ms`, or no | | `normalizeAnalyzeOnSettle` | function | Normalize the two spellings of an analyst-on-settle entry to the route form. | | `observe` | function | The third-person trace analyst: read a worker's trace and produce steer findings for the next attempt plus durable `learned` facts for the cross-run corpus. | @@ -696,7 +694,6 @@ Import from `@tangle-network/agent-runtime/kernel` — 717 exports. | `settledToIteration` | function | The step-8 merge-boundary adapter (M4): rehydrate a `Settled.done` into the kernel's | | `settledWorkerOut` | function | What a settled worker exposes as its output artifact (the blob the brain's | | `spendFromUsageEvents` | function | Fold a normalized `UsageEvent` array into a `Spend`. Tokens and usd are separate | -| `steeringDriver` | function | Interpret a {@link SteeringDirectiveData} as a loop `Driver` — the ONE interpreter both | | `stopSentinel` | function | A unique, attributable stop sentinel for a node (ralph-loop style). Deterministic from the | | `streamAgentTurn` | function | Run ONE agent turn on any backend kind and stream its events. Yields the | | `streamRouterChatWithTools` | function | The SAME completion as `routerChatWithTools`, taken over SSE (`stream: true`) and reassembled | @@ -826,7 +823,6 @@ Import from `@tangle-network/agent-runtime/kernel` — 717 exports. | `DownMessageDeliveryAttempt` | interface | A durable marker written after authorization and immediately before Runtime calls `Scope.send`. | | `DownMessageEvent` | interface | A parent→child delivery result (the down-leg): recorded for observability, never pulled back by | | `DriveHarness` | interface | How to run an external harness as the DRIVER, with the coordination verbs mounted — the substrate | -| `DumbDriverOptions` | interface | Options for {@link dumbDriver}. | | `EdgeTraversal` | interface | One recorded edge traversal — the in-memory row; the journal twin is the `edge` SpawnEvent. | | `EqualKArm` | interface | One arm of an equal-k comparison — a labeled trajectory (a `TrajectoryReport` is one arm's whole | | `EqualKOnCostOptions` | interface | `equalKOnCost(arms, { tolerance? })` — assert arms are comparable at EQUAL conserved COST | @@ -876,7 +872,6 @@ Import from `@tangle-network/agent-runtime/kernel` — 717 exports. | `MaterializedExecutionIdentity` | interface | External execution identity that operators can use to join this node to its backend. | | `McpEndpoint` | interface | Where a handle's MCP server lives; headers carry per-artifact scoping. | | `MountManifestEntry` | interface | One mounted resource recorded during box preparation — a pure provenance | -| `NaiveDriverOptions` | interface | Options for {@link naiveDriver}. | | `NodeExecutionIdentity` | interface | Durable identity of one realized node. Missing digests mean the input was not canonical JSON. | | `NoWinnerError` | interface | A driver's `act()` rejection, normalized to a serializable triple so it survives the typed | | `OpenSandboxRunBeforeStartContext` | interface | Context available after the box/session exists and before the first prompt is | @@ -992,7 +987,6 @@ Import from `@tangle-network/agent-runtime/kernel` — 717 exports. | `AgentEnvironmentProviderRef` | type | Provider object or registry name accepted by runtime provider adapters. | | `AgentProfileRef` | type | Portable profile reference: inline profile or provider catalog id. | | `AgentTurnBackend` | type | The execution substrate one turn runs on — a closed discriminated union over | -| `ApplyContinuation` | type | Fold a steering string into the caller's Task shape, producing the Task for | | `AssertTraceDerivedFindings` | type | The firewall assertion contract, re-stated for the reactive seam (PORT of | | `AuthoredProfile` | type | What the supervisor AUTHORS per sub-task: one complete canonical profile whose name and | | `AuthorizeDownMessage` | type | Product decision over an exact continuation before it is durably recorded or delivered. | @@ -1049,8 +1043,6 @@ Import from `@tangle-network/agent-runtime/kernel` — 717 exports. | `SpawnEvent` | type | Journaled spawn-tree events (B1/B2). `seq` is the cursor order; `at` is an ISO | | `SpawnPrior` | type | What a KEYED spawn resolved to when the key had a prior attempt. Absent on a fresh key (and on | | `SpawnRejection` | type | Fail-closed spawn rejections: an exhausted pool, a dollar request against a root that budgets | -| `SteeringDecision` | type | Terminal-or-continue decision shared by all three steering drivers. The | -| `SteeringDirectiveData` | type | A steering POLICY as plain data — the delegates-edge directive form of the two control | | `StopDecision` | type | A stop rule's answer. `reason` is required when stopping — a run that ends must be able to say | | `StopRule` | type | Evaluated from the progress feed, never from the budget. Pure and synchronous: it is called on | | `StrategyMessage` | type | One provider-neutral conversation record carried between strategy shots. | diff --git a/docs/api/runtime.md b/docs/api/runtime.md index 6248383f..b0a5400d 100644 --- a/docs/api/runtime.md +++ b/docs/api/runtime.md @@ -2791,7 +2791,7 @@ Result export. Default: write `matrix-result.json` under the run dir and > `optional` **dispatch?**: `ProfileDispatchFn`\<[`LeaderboardScenario`](#leaderboardscenario)\<`TCase`\>, `TArtifact`\> LEVEL 2 — full dispatch replacement (in-process products bring their own). - The default is `loopDispatch` + the naive steering directive over the resolved backend. + The default is `loopDispatch` + the naive retry driver over the resolved backend. ##### judges? @@ -7351,94 +7351,6 @@ Kill every spawned server. Idempotent. *** -### NaiveDriverOptions - -Options for [naiveDriver](#naivedriver). - -#### Type Parameters - -##### Task - -`Task` - -#### Properties - -##### continuation - -> **continuation**: `string` - -The fixed continuation issued every round after shot 0. The same string is -sent whether the prior shot passed inspection or not — the naive driver -reads no part of the verdict. Domain text is the caller's; the substrate -supplies none. - -##### applyContinuation - -> **applyContinuation**: [`ApplyContinuation`](#applycontinuation)\<`Task`\> - -Folds `continuation` into the caller's Task shape for the next shot. - -##### maxIterations - -> **maxIterations**: `number` - -Hard shot cap. The loop stops refining once history reaches this length. - -##### name? - -> `optional` **name?**: `string` - -Trace-event identifier. Default `'naive'`. - -*** - -### DumbDriverOptions - -Options for [dumbDriver](#dumbdriver). - -#### Type Parameters - -##### Task - -`Task` - -#### Properties - -##### onPass - -> **onPass**: `string` - -Continuation issued when the prior shot's verdict is valid. In a -stop-on-pass loop this is rarely reached (a valid shot ends the loop), but -it is required so the driver is total over the pass/fail bit; pass a -confirmation/keep-going string. - -##### onFail - -> **onFail**: `string` - -Continuation issued when the prior shot's verdict is NOT valid. - -##### applyContinuation - -> **applyContinuation**: [`ApplyContinuation`](#applycontinuation)\<`Task`\> - -Folds the chosen continuation into the caller's Task shape. - -##### maxIterations - -> **maxIterations**: `number` - -Hard shot cap. The loop stops refining once history reaches this length. - -##### name? - -> `optional` **name?**: `string` - -Trace-event identifier. Default `'dumb'`. - -*** - ### AuthorStrategyOptions #### Properties @@ -14090,7 +14002,7 @@ Assignment identity within the parent manager; absent only for the root. ###### Inherited from -[`SupervisorNodeContext`](#supervisornodecontext).[`task`](#task-25) +[`SupervisorNodeContext`](#supervisornodecontext).[`task`](#task-22) ##### signal @@ -18822,110 +18734,6 @@ controls such as `timeoutMs`. *** -### SteeringDecision - -> **SteeringDecision** = `"refine"` \| `"pick-winner"` \| `"fail"` - -Terminal-or-continue decision shared by all three steering drivers. The -non-terminal `'refine'` keeps the loop running another shot; the terminal -`'pick-winner'`/`'fail'` stop it (`isTerminalDecision` in run-loop.ts treats -`'pick-winner'` and `'fail'` as terminal and any other string as a request -for another round). Identical to the reference refine driver's decision set. - -*** - -### SteeringDirectiveData - -> **SteeringDirectiveData** = \{ `kind`: `"naive"`; `continuation`: `string`; `maxTraversals`: `number`; \} \| \{ `kind`: `"dumb"`; `onPass`: `string`; `onFail`: `string`; `maxTraversals`: `number`; \} - -A steering POLICY as plain data — the delegates-edge directive form of the two control -drivers. `kind` names how much of the verdict the policy may read (the experimental axis); -the continuation strings are the payload. Because this is JSON-able data, it is versionable -in a prompt registry and attachable to a graph edge — the same policy that used to exist -only as a builder FUNCTION, which made it invisible to any optimizer. Default texts are -seeded in the kernel prompt registry (`delegates/naive-continuation`, -`delegates/dumb-continuation-pass` / `-fail`); a caller may carry its own. - -#### Union Members - -##### Type Literal - -\{ `kind`: `"naive"`; `continuation`: `string`; `maxTraversals`: `number`; \} - -###### kind - -> `readonly` **kind**: `"naive"` - -Reads NOTHING from the verdict: one fixed continuation every round. - -###### continuation - -> `readonly` **continuation**: `string` - -###### maxTraversals - -> `readonly` **maxTraversals**: `number` - -Hard traversal cap: the loop stops refining once history reaches this length. - -*** - -##### Type Literal - -\{ `kind`: `"dumb"`; `onPass`: `string`; `onFail`: `string`; `maxTraversals`: `number`; \} - -###### kind - -> `readonly` **kind**: `"dumb"` - -Reads ONLY `verdict.valid` (the boolean): one of two fixed continuations. - -###### onPass - -> `readonly` **onPass**: `string` - -###### onFail - -> `readonly` **onFail**: `string` - -###### maxTraversals - -> `readonly` **maxTraversals**: `number` - -*** - -### ApplyContinuation - -> **ApplyContinuation**\<`Task`\> = (`task`, `continuation`) => `Task` - -Fold a steering string into the caller's Task shape, producing the Task for -the next shot. The substrate never assumes how a Task carries its prompt, so -the caller supplies this — the same way it supplies `taskToPrompt`. The -original `task` is passed so the fold can preserve task-level fields (ids, -fixtures, feature names) and replace only the instruction. - -#### Type Parameters - -##### Task - -`Task` - -#### Parameters - -##### task - -`Task` - -##### continuation - -`string` - -#### Returns - -`Task` - -*** - ### ChampionPolicy > **ChampionPolicy** = `"score"` \| `"costAware"` @@ -22859,112 +22667,6 @@ refuses local processes. A profile with no MCP surface returns zero tools. *** -### steeringDriver() - -> **steeringDriver**\<`Task`, `Output`\>(`directive`, `applyContinuation`, `name?`): [`Driver`](index.md#driver)\<`Task`, `Output`, [`SteeringDecision`](#steeringdecision)\> - -Interpret a [SteeringDirectiveData](#steeringdirectivedata) as a loop `Driver` — the ONE interpreter both -control policies share. The directive is data; only `applyContinuation` (how the caller's -opaque Task carries a steering string) remains code, exactly as `taskToPrompt` does. - -#### Type Parameters - -##### Task - -`Task` - -##### Output - -`Output` - -#### Parameters - -##### directive - -[`SteeringDirectiveData`](#steeringdirectivedata) - -##### applyContinuation - -[`ApplyContinuation`](#applycontinuation)\<`Task`\> - -##### name? - -`string` - -#### Returns - -[`Driver`](index.md#driver)\<`Task`, `Output`, [`SteeringDecision`](#steeringdecision)\> - -*** - -### ~~naiveDriver()~~ - -> **naiveDriver**\<`Task`, `Output`\>(`options`): [`Driver`](index.md#driver)\<`Task`, `Output`, [`SteeringDecision`](#steeringdecision)\> - -Thin compatibility wrapper over [steeringDriver](#steeringdriver) for the naive (no-signal) control. - -#### Type Parameters - -##### Task - -`Task` - -##### Output - -`Output` - -#### Parameters - -##### options - -[`NaiveDriverOptions`](#naivedriveroptions)\<`Task`\> - -#### Returns - -[`Driver`](index.md#driver)\<`Task`, `Output`, [`SteeringDecision`](#steeringdecision)\> - -#### Deprecated - -The policy is DATA now — build the directive and interpret it: - `steeringDriver({ kind: 'naive', continuation, maxTraversals }, applyContinuation)`. This - wrapper survives for existing callers and will be removed in the next major. - -*** - -### ~~dumbDriver()~~ - -> **dumbDriver**\<`Task`, `Output`\>(`options`): [`Driver`](index.md#driver)\<`Task`, `Output`, [`SteeringDecision`](#steeringdecision)\> - -Thin compatibility wrapper over [steeringDriver](#steeringdriver) for the dumb (pass/fail-only) control. - -#### Type Parameters - -##### Task - -`Task` - -##### Output - -`Output` - -#### Parameters - -##### options - -[`DumbDriverOptions`](#dumbdriveroptions)\<`Task`\> - -#### Returns - -[`Driver`](index.md#driver)\<`Task`, `Output`, [`SteeringDecision`](#steeringdecision)\> - -#### Deprecated - -The policy is DATA now — build the directive and interpret it: - `steeringDriver({ kind: 'dumb', onPass, onFail, maxTraversals }, applyContinuation)`. This - wrapper survives for existing callers and will be removed in the next major. - -*** - ### assertStrategyContract() > **assertStrategyContract**(`code`): `void` diff --git a/docs/canonical-api.md b/docs/canonical-api.md index 457fe523..f82ea551 100644 --- a/docs/canonical-api.md +++ b/docs/canonical-api.md @@ -4,7 +4,7 @@ Generated signatures and the complete export list live in docs/api/. Run pnpm docs:freshness after editing this file. --> -> **Version 0.124.0.** +> **Version 0.125.0.** > [`docs/api/primitive-catalog.md`](./api/primitive-catalog.md) lists every export and import path. > `agent-eval` must satisfy `>=0.142.2 <0.143.0`. > `sandbox` must satisfy `>=0.17.2 <0.18.0`. diff --git a/package.json b/package.json index 40714eb8..d7a2561d 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@tangle-network/agent-runtime", - "version": "0.124.0", + "version": "0.125.0", "description": "Shared task-lifecycle skeleton for agents: a recursive loop kernel for chat turns, one-shot tasks, and multi-attempt loops, with trace capture and eval-gated self-improvement. Domain behavior lives in adapters; scoring and ship-gates in @tangle-network/agent-eval.", "homepage": "https://github.com/tangle-network/agent-runtime#readme", "repository": { diff --git a/src/runtime/define-leaderboard.test.ts b/src/runtime/define-leaderboard.test.ts index e9dce037..fded59a0 100644 --- a/src/runtime/define-leaderboard.test.ts +++ b/src/runtime/define-leaderboard.test.ts @@ -1,7 +1,7 @@ import { mkdtempSync } from 'node:fs' import { tmpdir } from 'node:os' import { join } from 'node:path' -import { HARNESS_NATIVE_MODEL } from '@tangle-network/agent-eval' +import { type DefaultVerdict, HARNESS_NATIVE_MODEL } from '@tangle-network/agent-eval' import type { AgentProfile } from '@tangle-network/agent-interface' import type { CreateSandboxOptions, SandboxEvent } from '@tangle-network/sandbox' import { describe, expect, it } from 'vitest' @@ -9,8 +9,11 @@ import { defineLeaderboard, type LeaderboardIterationInfo, type LeaderboardRunContext, + type LeaderboardScenario, + naiveRetryDriver, } from './define-leaderboard' import { inProcessSandboxClient } from './in-process-sandbox-client' +import type { Iteration } from './types' interface FakeCase { id: string @@ -315,3 +318,71 @@ describe('defineLeaderboard', () => { await expect(dup.preflight()).rejects.toThrow(/duplicate case id/) }) }) + +// The no-signal retry floor: shot-0 plan, verbatim re-run, leak-free firewall. +// surviving loop-kernel steering control, and these tests hold the same observable contract its +// predecessor's suite held — shot-0 plan, verbatim re-run, the leak-free firewall (reads only +// `verdict.valid`), stop-on-valid, stop-at-cap, and the refine/pick-winner/fail decision set. +describe('naiveRetryDriver — the no-signal retry floor', () => { + type Case = { id: string; answer: string } + type Task = LeaderboardScenario + const task: Task = { id: 't', kind: 'leaderboard-case', case: { id: 't', answer: 'A' } } + const driver = naiveRetryDriver(3) + + function iter(verdict: DefaultVerdict | undefined, index = 0): Iteration { + return { + index, + task, + agentRunName: 'agent', + output: 'out', + verdict, + events: [], + startedAt: 0, + endedAt: 1, + costUsd: 0, + tokenUsage: { input: 0, output: 0 }, + } + } + + it('runs the original task at shot 0', async () => { + expect(await driver.plan(task, [])).toEqual([task]) + }) + + it('re-runs the SAME task verbatim, reading only verdict.valid (leak-free firewall)', async () => { + // Tripwire: getters on `notes`/`scores` throw if the driver reads grader findings. + const trapVerdict = { valid: false, score: 0 } as { valid: boolean; score: number } + Object.defineProperty(trapVerdict, 'notes', { + get() { + throw new Error('naiveRetryDriver read verdict.notes — firewall breached') + }, + }) + Object.defineProperty(trapVerdict, 'scores', { + get() { + throw new Error('naiveRetryDriver read verdict.scores — firewall breached') + }, + }) + const planned = await driver.plan(task, [iter(trapVerdict)]) + expect(planned).toEqual([task]) + expect(planned[0]).toBe(task) + // A verdict-free history plans identically: a missing verdict is not-valid, never a throw. + expect(await driver.plan(task, [iter(undefined)])).toEqual([task]) + }) + + it('stops on a valid shot and at the shot cap', async () => { + expect(await driver.plan(task, [iter({ valid: true, score: 1 })])).toEqual([]) + const capped = [iter(undefined, 0), iter(undefined, 1), iter(undefined, 2)] + expect(await driver.plan(task, capped)).toEqual([]) + }) + + it('decides pick-winner when any shot is valid', () => { + expect( + driver.decide([iter({ valid: false, score: 0 }), iter({ valid: true, score: 1 }, 1)]), + ).toBe('pick-winner') + }) + + it('decides refine while under the cap, fail at the cap', () => { + expect(driver.decide([iter({ valid: false, score: 0 })])).toBe('refine') + const capped = [iter(undefined, 0), iter(undefined, 1), iter(undefined, 2)] + expect(driver.decide(capped)).toBe('fail') + }) +}) diff --git a/src/runtime/define-leaderboard.ts b/src/runtime/define-leaderboard.ts index 78063f71..bba6127e 100644 --- a/src/runtime/define-leaderboard.ts +++ b/src/runtime/define-leaderboard.ts @@ -3,7 +3,7 @@ * * A product's harness×model leaderboard is always the same assembly: expand a * base profile across the harness×model axes (`expandProfileAxes`), run every - * (profile, case) cell as a driven loop (`loopDispatch` + the naive steering directive), score + * (profile, case) cell as a driven loop (`loopDispatch` + the naive retry driver), score * with the domain's grader, and emit ONE `runProfileMatrix` call. Each product * hand-rolled that assembly (~650 lines each) and re-hit the same footguns: * stale cell-cache reuse, zero-token stub cells, missing model snapshots. @@ -53,9 +53,8 @@ import { collectAgentResponseText, type SandboxEvent } from '@tangle-network/san import { leaderboard, renderLeaderboardMarkdown } from './benchmark-report' import { loopDispatch } from './loop-dispatch' import { resolveSandboxClient } from './resolve-sandbox-client' -import { type SteeringDecision, steeringDriver } from './steering-drivers' import { isHarnessNativeModel } from './supervise/model-policy' -import type { LoopResult, SandboxClient } from './types' +import type { Driver, LoopResult, SandboxClient } from './types' /** Structured per-case verdict a `score` function may return (a bare number is * shorthand for `{ composite }`). `composite` is the [0,1] leaderboard score; @@ -72,6 +71,42 @@ export interface LeaderboardScenario extends Scenario { case: TCase } +/** Terminal-or-continue decision set of {@link naiveRetryDriver}: the non-terminal `'refine'` + * requests another shot; `'pick-winner'`/`'fail'` stop the loop (`isTerminalDecision` in + * run-loop.ts treats them as terminal). */ +export type NaiveRetryDecision = 'refine' | 'pick-winner' | 'fail' + +/** + * The no-signal retry floor as a bare `Driver` literal: re-run the case VERBATIM as an + * independent attempt until one shot scores (`verdict.valid`) or the shot cap. It reads ONLY + * `verdict.valid` — never `notes`/`scores` — so a leaderboard cell's retries carry no grader + * findings into the next shot (the leak-free firewall). Steering POLICY texts are registry data + * (`delegates/naive-continuation` and siblings in the kernel prompt registry) attached to graph + * edges; this loop-kernel control folds no text in — each shot is the same task object, and the + * per-shot nonce in `taskToPrompt` keeps the attempts independent at the router. + */ +export function naiveRetryDriver( + shots: number, +): Driver, TArtifact, NaiveRetryDecision> { + return { + name: 'naive', + plan(task, history) { + if (history.length === 0) return Promise.resolve([task]) + const last = history[history.length - 1] + if (last?.verdict?.valid === true) return Promise.resolve([]) + if (history.length >= shots) return Promise.resolve([]) + return Promise.resolve([task]) + }, + decide(history) { + if (history.some((it) => it.verdict?.valid)) return 'pick-winner' + return history.length < shots ? 'refine' : 'fail' + }, + describePlan() { + return { kind: 'refine', rationale: 'naive fixed continuation (no grade signal)' } + }, + } +} + /** One extra CLI flag a spec declares. Parsed by `run()` as `-- ` * and surfaced to every hook via `ctx.args`. */ export interface LeaderboardFlagSpec { @@ -221,7 +256,7 @@ export interface LeaderboardSpec { ctx: LeaderboardRunContext, ) => Promise | void /** LEVEL 2 — full dispatch replacement (in-process products bring their own). - * The default is `loopDispatch` + the naive steering directive over the resolved backend. */ + * The default is `loopDispatch` + the naive retry driver over the resolved backend. */ dispatch?: ProfileDispatchFn, TArtifact> /** LEVEL 2 — full judge replacement. Default: `score` wrapped as one judge. */ judges?: JudgeConfig>[] @@ -485,7 +520,7 @@ export function defineLeaderboard( const cellDispatch = loopDispatch< LeaderboardScenario, TArtifact, - SteeringDecision, + NaiveRetryDecision, LeaderboardScenario, TArtifact >({ @@ -521,16 +556,10 @@ export function defineLeaderboard( : {}), } return { - // The naive steering directive = the no-signal retry floor: re-run the same case as - // an independent attempt until one scores (>0) or the shot cap. The policy is data - // (`SteeringDirectiveData`); the task is re-run verbatim (identity applyContinuation), - // and the continuation is EMPTY so the no-op stays a no-op even if a future - // applyContinuation ever folds the text in. - driver: steeringDriver, TArtifact>( - { kind: 'naive', continuation: '', maxTraversals: shots }, - (task) => task, - 'naive', - ), + // The no-signal retry floor: re-run the same case as an independent attempt until + // one scores (>0) or the shot cap. The task is re-run verbatim — no grader findings + // cross into the next shot. + driver: naiveRetryDriver(shots), agentRun: { profile: cellProfile, taskToPrompt: (s) => `${promptOf(s)}\n\n`, diff --git a/src/runtime/index.ts b/src/runtime/index.ts index a6ac1b4a..d8b20abd 100644 --- a/src/runtime/index.ts +++ b/src/runtime/index.ts @@ -136,7 +136,7 @@ export { stopSentinel, } from './completion' // The declarative eval-leaderboard facade: cases + prompt + score → one -// runProfileMatrix call (expandProfileAxes × loopDispatch × naiveDriver), +// runProfileMatrix call (expandProfileAxes × loopDispatch × the naive retry driver), // with a structural BenchmarkAdapter view via toBenchmarkAdapter(). export { type DefinedLeaderboard, @@ -419,16 +419,6 @@ export { type StdioMcpConnection, type StdioMcpServerSpec, } from './stdio-mcp-client' -export { - type ApplyContinuation, - type DumbDriverOptions, - dumbDriver, - type NaiveDriverOptions, - naiveDriver, - type SteeringDecision, - type SteeringDirectiveData, - steeringDriver, -} from './steering-drivers' // The optimization suite: a domain = an Environment (5 hooks); a Strategy = how the // budget is spent to beat its check. Built-ins `sample`/`refine`; author your own with // `defineStrategy` (compose shot() + critique(), zero Supervisor ceremony); compare diff --git a/src/runtime/steering-drivers.test.ts b/src/runtime/steering-drivers.test.ts deleted file mode 100644 index 19d408d0..00000000 --- a/src/runtime/steering-drivers.test.ts +++ /dev/null @@ -1,126 +0,0 @@ -import type { DefaultVerdict } from '@tangle-network/agent-eval' -import { describe, expect, it } from 'vitest' -import { dumbDriver, naiveDriver } from './steering-drivers' -import type { Iteration } from './types' - -// A Task shape WITHOUT a `prompt` field, to prove the substrate builders make -// no assumption about how a Task carries its instruction: the caller's -// `applyContinuation` owns that fold. -interface Task { - id: string - instruction: string -} -type Output = { text: string } - -const applyContinuation = (task: Task, continuation: string): Task => ({ - ...task, - instruction: continuation, -}) - -function iter(verdict: DefaultVerdict | undefined, index = 0): Iteration { - return { - index, - task: { id: 't', instruction: 'orig' }, - agentRunName: 'agent', - output: { text: 'out' }, - verdict, - events: [], - startedAt: 0, - endedAt: 1, - costUsd: 0, - tokenUsage: { input: 0, output: 0 }, - } -} - -const task: Task = { id: 't', instruction: 'do the thing' } - -describe('naiveDriver — no-signal control', () => { - const driver = naiveDriver({ - continuation: 'keep going', - applyContinuation, - maxIterations: 3, - }) - - it('runs the original task at shot 0', async () => { - expect(await driver.plan(task, [])).toEqual([task]) - }) - - it('issues the fixed continuation regardless of the verdict findings', async () => { - // A failing verdict packed with rich findings the driver MUST ignore. - const failed = iter({ - valid: false, - score: 0.1, - notes: 'use foo() not bar()', - scores: { x: 0.1 }, - }) - const planned = await driver.plan(task, [failed]) - expect(planned).toEqual([{ id: 't', instruction: 'keep going' }]) - // The continuation is independent of the verdict: a verdict-free history - // yields the identical plan. - const noVerdict = await driver.plan(task, [iter(undefined)]) - expect(noVerdict).toEqual([{ id: 't', instruction: 'keep going' }]) - }) - - it('stops on a valid shot and at the shot cap', async () => { - expect(await driver.plan(task, [iter({ valid: true, score: 1 })])).toEqual([]) - const capped = [iter(undefined, 0), iter(undefined, 1), iter(undefined, 2)] - expect(await driver.plan(task, capped)).toEqual([]) - }) -}) - -describe('dumbDriver — pass/fail-only control (leak-free firewall)', () => { - const driver = dumbDriver({ - onPass: 'looks good, confirm', - onFail: 'failed, try again', - applyContinuation, - maxIterations: 3, - }) - - it('reads ONLY verdict.valid, never notes/scores', async () => { - // Tripwire: a getter on `notes`/`scores` throws if the driver touches them. - const trapVerdict = { valid: false, score: 0 } as DefaultVerdict - Object.defineProperty(trapVerdict, 'notes', { - get() { - throw new Error('dumbDriver read verdict.notes — firewall breached') - }, - }) - Object.defineProperty(trapVerdict, 'scores', { - get() { - throw new Error('dumbDriver read verdict.scores — firewall breached') - }, - }) - const planned = await driver.plan(task, [iter(trapVerdict)]) - expect(planned).toEqual([{ id: 't', instruction: 'failed, try again' }]) - }) - - it('treats a missing/undefined verdict as not-valid (total over the bit)', async () => { - expect(await driver.plan(task, [iter(undefined)])).toEqual([ - { id: 't', instruction: 'failed, try again' }, - ]) - }) - - it('stops the loop on a valid shot', async () => { - expect(await driver.plan(task, [iter({ valid: true, score: 1 })])).toEqual([]) - }) -}) - -describe('shared decide() — terminates exactly like refine', () => { - const driver = dumbDriver({ - onPass: 'p', - onFail: 'f', - applyContinuation, - maxIterations: 3, - }) - - it('pick-winner when any shot is valid', () => { - expect( - driver.decide([iter({ valid: false, score: 0 }), iter({ valid: true, score: 1 }, 1)]), - ).toBe('pick-winner') - }) - - it('refine while under the cap, fail at the cap', () => { - expect(driver.decide([iter({ valid: false, score: 0 })])).toBe('refine') - const capped = [iter(undefined, 0), iter(undefined, 1), iter(undefined, 2)] - expect(driver.decide(capped)).toBe('fail') - }) -}) diff --git a/src/runtime/steering-drivers.ts b/src/runtime/steering-drivers.ts deleted file mode 100644 index 35558e76..00000000 --- a/src/runtime/steering-drivers.ts +++ /dev/null @@ -1,238 +0,0 @@ -/** - * - * Leak-free steering drivers — the non-LLM controls for the driven loop. - * - * These are the two sibling `Driver`s of the `refine` reference driver - * (examples/driver-loop/driver-loop.ts), differing ONLY in how much of the - * prior `verdict` their `plan()` is allowed to read. The amount of the verdict - * a driver reads IS the experimental axis: - * - * • refine — reads `verdict.notes` (and/or `verdict.scores`): the grader's - * findings. A reviewer/judge LLM turns those findings into the - * next prompt. This is the "coached" condition. - * • dumb — reads ONLY `verdict.valid` (the pass/fail boolean). No findings, - * no scores, no LLM. The next prompt is one of two fixed strings - * keyed on pass/fail. This is the leak-free pass/fail control. - * • naive — reads NOTHING from the verdict. The next prompt is one fixed - * continuation string every round. This is the leak-free - * no-signal control. - * - * The `dumb → refine` gap measured on the same task therefore isolates how much - * the grader's findings (its notes/scores) inflate the result beyond a bare - * pass/fail signal: it is exactly the delta between reading `verdict.valid` and - * reading `verdict.notes`. The `naive → dumb` gap isolates the value of the - * pass/fail bit alone. Run all three against one task to attribute a loop's - * lift to its actual source instead of crediting it to the grader for free. - * - * GENERALITY CONTRACT - * ------------------- - * These builders carry ZERO domain coupling: - * • The continuation strings are PARAMETERS. The caller passes its own text; - * the substrate hardcodes none. - * • The Task shape is OPAQUE. The builder never assumes a `prompt` field (or - * any field). The caller supplies `applyContinuation(task, text) → Task`, - * which folds a steering string into the caller's own Task shape — exactly - * symmetric with the caller-supplied `AgentRunSpec.taskToPrompt`. For a Task - * that is `{ prompt: string; … }`, the fold is the one-liner - * `(task, text) => ({ ...task, prompt: text })`. - * - * They consume nothing the loop kernel does not already give `plan()`/`decide()` - * — `runAgentRounds` requires no change. A benchmark picks a driver by name and the - * kernel does the rest. - * - * @experimental - */ - -import type { Driver, Iteration } from './types' - -/** - * Terminal-or-continue decision shared by all three steering drivers. The - * non-terminal `'refine'` keeps the loop running another shot; the terminal - * `'pick-winner'`/`'fail'` stop it (`isTerminalDecision` in run-loop.ts treats - * `'pick-winner'` and `'fail'` as terminal and any other string as a request - * for another round). Identical to the reference refine driver's decision set. - */ -export type SteeringDecision = 'refine' | 'pick-winner' | 'fail' - -/** - * A steering POLICY as plain data — the delegates-edge directive form of the two control - * drivers. `kind` names how much of the verdict the policy may read (the experimental axis); - * the continuation strings are the payload. Because this is JSON-able data, it is versionable - * in a prompt registry and attachable to a graph edge — the same policy that used to exist - * only as a builder FUNCTION, which made it invisible to any optimizer. Default texts are - * seeded in the kernel prompt registry (`delegates/naive-continuation`, - * `delegates/dumb-continuation-pass` / `-fail`); a caller may carry its own. - */ -export type SteeringDirectiveData = - | { - /** Reads NOTHING from the verdict: one fixed continuation every round. */ - readonly kind: 'naive' - readonly continuation: string - /** Hard traversal cap: the loop stops refining once history reaches this length. */ - readonly maxTraversals: number - } - | { - /** Reads ONLY `verdict.valid` (the boolean): one of two fixed continuations. */ - readonly kind: 'dumb' - readonly onPass: string - readonly onFail: string - readonly maxTraversals: number - } - -/** - * Interpret a {@link SteeringDirectiveData} as a loop `Driver` — the ONE interpreter both - * control policies share. The directive is data; only `applyContinuation` (how the caller's - * opaque Task carries a steering string) remains code, exactly as `taskToPrompt` does. - */ -export function steeringDriver( - directive: SteeringDirectiveData, - applyContinuation: ApplyContinuation, - name?: string, -): Driver { - const driverName = name ?? directive.kind - const maxIterations = directive.maxTraversals - const continuationFor = (passed: boolean): string => - directive.kind === 'naive' - ? directive.continuation - : passed - ? directive.onPass - : directive.onFail - const rationale = - directive.kind === 'naive' - ? 'naive fixed continuation (no grade signal)' - : 'dumb pass/fail-only continuation (no grader findings)' - return { - name: driverName, - plan(task, history) { - if (history.length === 0) return Promise.resolve([task]) - const last = history[history.length - 1] - const passed = last?.verdict?.valid === true - if (passed) return Promise.resolve([]) - if (history.length >= maxIterations) return Promise.resolve([]) - // The ONLY verdict read in the steering path is the pass/fail boolean, and only for the - // dumb directive. `.notes`/`.scores` are never touched — the leak-free firewall. - return Promise.resolve([applyContinuation(task, continuationFor(passed))]) - }, - decide(history) { - return decideUntilValidOrCapped(history, maxIterations) - }, - describePlan() { - return { kind: 'refine', rationale } - }, - } -} - -/** - * Fold a steering string into the caller's Task shape, producing the Task for - * the next shot. The substrate never assumes how a Task carries its prompt, so - * the caller supplies this — the same way it supplies `taskToPrompt`. The - * original `task` is passed so the fold can preserve task-level fields (ids, - * fixtures, feature names) and replace only the instruction. - */ -export type ApplyContinuation = (task: Task, continuation: string) => Task - -/** - * Shared `decide()` for the steering drivers. Verbatim semantics of the - * reference refine driver: a valid iteration anywhere → terminal `'pick-winner'`; - * otherwise `'refine'` (non-terminal, run another shot) while history is under - * the shot cap, else terminal `'fail'`. Pure over history; reads only - * `verdict.valid` (never notes/scores) so it adds no leak of its own. - */ -function decideUntilValidOrCapped( - history: ReadonlyArray>, - maxIterations: number, -): SteeringDecision { - if (history.some((it) => it.verdict?.valid)) return 'pick-winner' - return history.length < maxIterations ? 'refine' : 'fail' -} - -/** Options for {@link naiveDriver}. */ -export interface NaiveDriverOptions { - /** - * The fixed continuation issued every round after shot 0. The same string is - * sent whether the prior shot passed inspection or not — the naive driver - * reads no part of the verdict. Domain text is the caller's; the substrate - * supplies none. - */ - continuation: string - /** Folds `continuation` into the caller's Task shape for the next shot. */ - applyContinuation: ApplyContinuation - /** Hard shot cap. The loop stops refining once history reaches this length. */ - maxIterations: number - /** Trace-event identifier. Default `'naive'`. */ - name?: string -} - -/** - * `naiveDriver` — the no-signal steering control. - * - * `plan()` runs the initial `task` at shot 0, then issues the SAME fixed - * `continuation` every subsequent round until a shot is valid or the cap is - * hit. It reads NOTHING from `history[last].verdict` — not `.valid`, not - * `.notes`, not `.scores`. It is the floor a coached loop must beat to earn its - * coaching: any lift over naive that is not also present in `dumb` is - * attributable to the pass/fail bit, and any lift of `refine` over `dumb` is - * attributable to the grader's findings. - */ -/** Thin compatibility wrapper over {@link steeringDriver} for the naive (no-signal) control. - * @deprecated The policy is DATA now — build the directive and interpret it: - * `steeringDriver({ kind: 'naive', continuation, maxTraversals }, applyContinuation)`. This - * wrapper survives for existing callers and will be removed in the next major. */ -export function naiveDriver( - options: NaiveDriverOptions, -): Driver { - const { continuation, applyContinuation, maxIterations, name = 'naive' } = options - return steeringDriver( - { kind: 'naive', continuation, maxTraversals: maxIterations }, - applyContinuation, - name, - ) -} - -/** Options for {@link dumbDriver}. */ -export interface DumbDriverOptions { - /** - * Continuation issued when the prior shot's verdict is valid. In a - * stop-on-pass loop this is rarely reached (a valid shot ends the loop), but - * it is required so the driver is total over the pass/fail bit; pass a - * confirmation/keep-going string. - */ - onPass: string - /** Continuation issued when the prior shot's verdict is NOT valid. */ - onFail: string - /** Folds the chosen continuation into the caller's Task shape. */ - applyContinuation: ApplyContinuation - /** Hard shot cap. The loop stops refining once history reaches this length. */ - maxIterations: number - /** Trace-event identifier. Default `'dumb'`. */ - name?: string -} - -/** - * `dumbDriver` — the pass/fail-only steering control. - * - * `plan()` runs the initial `task` at shot 0, then reads ONLY - * `history[last].verdict.valid` (the boolean) and issues `onPass` or `onFail` - * accordingly. It MUST NOT read `.notes` or `.scores` — that boundary is the - * leak-free firewall. A `verdict` with no `valid` set (or no verdict) is - * treated as not-valid, so the driver is total and never throws on a - * grader/transport gap. - * - * The `dumb → refine` gap is the headline measurement: refine reads the - * grader's `notes`, dumb reads only the pass/fail bit, so the difference is - * exactly the value the findings add over a bare boolean. - */ -/** Thin compatibility wrapper over {@link steeringDriver} for the dumb (pass/fail-only) control. - * @deprecated The policy is DATA now — build the directive and interpret it: - * `steeringDriver({ kind: 'dumb', onPass, onFail, maxTraversals }, applyContinuation)`. This - * wrapper survives for existing callers and will be removed in the next major. */ -export function dumbDriver( - options: DumbDriverOptions, -): Driver { - const { onPass, onFail, applyContinuation, maxIterations, name = 'dumb' } = options - return steeringDriver( - { kind: 'dumb', onPass, onFail, maxTraversals: maxIterations }, - applyContinuation, - name, - ) -} diff --git a/src/testing/fixtures/agent-improvement-proposal.json b/src/testing/fixtures/agent-improvement-proposal.json index 0e39fdda..a866cbfc 100644 --- a/src/testing/fixtures/agent-improvement-proposal.json +++ b/src/testing/fixtures/agent-improvement-proposal.json @@ -1,6 +1,6 @@ { "changedSurfaces": ["prompt"], - "digest": "sha256:2e3f72e29ef32b74e83428b297609ad49742c70e78ee18d3a30ef93f1270889f", + "digest": "sha256:e82ee0ae5974b86caf61ed711ba81e65cde04fab2d6eb2311d1983739e1114e1", "evaluation": { "decision": { "contributingChecks": [ @@ -4810,7 +4810,7 @@ ], "metadata": { "fixture": "agent-improvement-proposal", - "runtimeVersion": "0.124.0" + "runtimeVersion": "0.125.0" }, "objectives": [ { @@ -4921,8 +4921,8 @@ "baselineContentHash": "sha256:5c21ee53e513fc604cb09754e21c392b24a424da0ef37dbf8f1ee4a8a0b08f09", "candidateContentHash": "sha256:60fcbb1c728194bd51d7d19cb732d1c3f1881dce7e0a6266b41c8b98cfd65693", "kind": "agent-eval-loop", - "recordDigest": "sha256:c0025edf0ab73091b6d740c534ed4c3b4b660f90e82c4fadf0dd65fcd3a50d76", - "runId": "agent-runtime-0.124.0-proposal-fixture", + "recordDigest": "sha256:c13929ce670ba5b91cfc830c9e661cab4446293c3bfc6b7528d37e83bb71b314", + "runId": "agent-runtime-0.125.0-proposal-fixture", "schema": "agent-candidate-experiment" } }, @@ -4949,5 +4949,5 @@ ], "kind": "agent-improvement-proposal", "proposedAt": "2026-07-10T01:00:00.000Z", - "runId": "agent-runtime-0.124.0-proposal-fixture" + "runId": "agent-runtime-0.125.0-proposal-fixture" } diff --git a/src/testing/fixtures/agent-profile-improvement-proposal.json b/src/testing/fixtures/agent-profile-improvement-proposal.json index d9d0b85c..8bb1baa9 100644 --- a/src/testing/fixtures/agent-profile-improvement-proposal.json +++ b/src/testing/fixtures/agent-profile-improvement-proposal.json @@ -1,6 +1,6 @@ { "changedSurfaces": ["prompt", "skills"], - "digest": "sha256:cf9cab31f85543da571017e87c5cca7aa07656acd704d515d7832754f3ca4aec", + "digest": "sha256:f65828ac5a001185dd8d07206acfab2729a25f8e21e8500c9e087c5ea60003db", "evaluation": { "decision": { "contributingChecks": [ @@ -1715,7 +1715,7 @@ ], "metadata": { "fixture": "agent-profile-improvement-proposal", - "runtimeVersion": "0.124.0" + "runtimeVersion": "0.125.0" }, "objectives": [ { @@ -1826,7 +1826,7 @@ "baselineContentHash": "sha256:21c495a37c418c10bde64fbaa188beddeed31f1f051ea60a6a6582a9ee0db704", "candidateContentHash": "sha256:103f77bc8481601eef1ad5fe6ba84a40dffabc3a44f421f8c8559121edab84e9", "kind": "agent-eval-loop", - "recordDigest": "sha256:5d84308cbe6eb3596d9d3e55604d8830946622e922c5be16f81442f6b7cdf824", + "recordDigest": "sha256:ac56c24b908cbe377729646d7bcdb3d78a959ceb2a4f55134dcac52dc4b35e82", "runId": "profile-improvement-1", "schema": "agent-profile-improvement-experiment" }