From e48794a70c3715faa1eaba5da1111bc5e4bf1fec Mon Sep 17 00:00:00 2001 From: Drew Stone Date: Mon, 3 Aug 2026 00:46:05 -0600 Subject: [PATCH 1/2] =?UTF-8?q?feat(skills):=20codemode=20=E2=80=94=20conv?= =?UTF-8?q?ert=20ideas=20and=20loose=20context=20into=20agent=20graphs?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The graph layer shipped in 0.123.0 with zero consumers outside its own tests and examples -- the classic speculative-abstraction profile until something adopts it. This skill is the adoption surface: given to any agent (org skill dirs, or inline via profile.resources.skills, which has no size cap), it carries the authoring contract, the dialect routing (static graph vs dynamic-workflows script vs no orchestration at all), the measured budget floors, and the post-run evidence discipline: a graph worked only if its ledger and deliverable verdict say so. Every pothole in the skill is a failure mode proven from source or from live runs, not invented: analyzes-over-root validating but never firing, analyzes caps stopping nothing, refused spawns burning traversals, the mission belonging in deliverable.describe, perWorker defaulting to pool/4 under the 31,211 pi floor. IMPROVE.md maps self-improvement of this skill onto existing agent-eval machinery only -- skillOptOptimizationMethod already takes skill text as its surface, runImprovementLoop already gates promotion on a disjoint holdout -- with the loop owning exactly two closures and the eight seeded cases. Loose briefs on purpose: loose-context-in is the claim under test. --- skills/codemode/IMPROVE.md | 40 ++++++++ skills/codemode/SKILL.md | 95 +++++++++++++++++++ .../codemode/cases/cap-as-stop-mistake.json | 8 ++ skills/codemode/cases/floor-trap-pi.json | 11 +++ .../cases/mission-in-deliverable.json | 8 ++ skills/codemode/cases/review-pipeline.json | 14 +++ .../cases/runtime-discovered-fanout.json | 8 ++ .../codemode/cases/single-agent-suffices.json | 7 ++ .../codemode/cases/steer-heavy-drafting.json | 9 ++ skills/codemode/cases/unmeasured-harness.json | 9 ++ 10 files changed, 209 insertions(+) create mode 100644 skills/codemode/IMPROVE.md create mode 100644 skills/codemode/SKILL.md create mode 100644 skills/codemode/cases/cap-as-stop-mistake.json create mode 100644 skills/codemode/cases/floor-trap-pi.json create mode 100644 skills/codemode/cases/mission-in-deliverable.json create mode 100644 skills/codemode/cases/review-pipeline.json create mode 100644 skills/codemode/cases/runtime-discovered-fanout.json create mode 100644 skills/codemode/cases/single-agent-suffices.json create mode 100644 skills/codemode/cases/steer-heavy-drafting.json create mode 100644 skills/codemode/cases/unmeasured-harness.json diff --git a/skills/codemode/IMPROVE.md b/skills/codemode/IMPROVE.md new file mode 100644 index 00000000..03865880 --- /dev/null +++ b/skills/codemode/IMPROVE.md @@ -0,0 +1,40 @@ +# Improving the codemode skill: the loop, mapped onto what exists + +The improving artifact is `SKILL.md`'s text. +Nothing below is a new framework; every step names the existing agent-eval primitive it composes, per the adopt-or-improve rule. +The only code this loop owns is two closures and a case set — the slots the machinery deliberately leaves to the caller. + +## The loop + +``` +case (idea brief) ──► author agent + skill-vN ──► graph ──► runGraph OFFLINE ──► score ──► revise skill ──► gate ──► vN+1 +``` + +| Step | Primitive | What the loop supplies | +| --- | --- | --- | +| Skill text as candidate surface | `MutableSurface = string` (`campaign/types.ts:210`); read `SKILL.md` → string | one line | +| Generate graph from case | caller-owned `dispatchWithSurface(surface, scenario, ctx)` in `runCampaign` | **closure A**: run an author agent carrying skill-vN + the case brief; return the authored graph module | +| Execute offline | `runGraph` with scripted `brain`, stub leaf seam, in-memory journal/blobs (the `examples/graphs/` pattern) | part of closure A | +| Deterministic scoring | a `JudgeConfig` closure (the `golden-matcher`/`completion-verifier` pattern) | **closure B**: score from `GraphResult` — validation passed; expected edges present with >0 traversals; ledger outcomes match the case's expectations; `exhaustedEdges` empty unless expected; deliverable verdict correct on both a passing and a failing scripted run | +| Semantic scoring (only what mechanics can't see) | `judge-panel.ts` `ensembleJudge` (cross-family, fail-loud) | rubric: role decomposition quality, directive clarity | +| Revision | `skillOptOptimizationMethod` (requires a **string** surface — skill text is first-class) or `gepa-optimization-method`; trace-conditioned diffs via `reflective-mutation.ts` | config only | +| Generations + incumbent | `runOptimization` (retains every generation's surfaces and campaigns) | config only | +| Gated promotion | `runImprovementLoop` — disjoint train/holdout enforced, no-op winner forced to hold, `autoOnPromote: 'pr'` writes the winner back as a PR | config only | +| Audit trail | `search-ledger.ts` hash-chained JSONL | free | + +## Cases + +`cases/` seeds eight idea-briefs, each with `expect`: the edges a correct graph must have, ledger outcomes, whether analysts are warranted, and a floor-trap flag (the case is under-budgetable and a correct author must budget above the floor). +Case briefs are deliberately loose — "loose context in, correct graph out" is the skill's whole claim, so tidy specs would test the wrong thing. + +Holdout discipline: at least 3 of the 8 held out, never trained on; `runImprovementLoop` throws on overlap. + +## What is deliberately NOT built + +- No graph-diff scorer beyond the ledger checks — a graph is correct if it *runs* correctly offline, not if it textually matches a golden. +- No new optimizer, campaign runner, judge plumbing, or ledger — all named above. +- No live-backend scoring in the loop. Live runs are pursuit work, not skill-improvement work; the loop stays offline and free. + +## Known upstream gap this loop will hit + +`OptimizationMethodResult` returns `winnerSurface` only — full candidate history is an owed upstream extension (recorded in discovery docs 22/25). Workaround needing no code: `runOptimization` already retains every generation's surfaces. diff --git a/skills/codemode/SKILL.md b/skills/codemode/SKILL.md new file mode 100644 index 00000000..2c543d35 --- /dev/null +++ b/skills/codemode/SKILL.md @@ -0,0 +1,95 @@ +--- +name: codemode +description: "Convert an idea, workflow, or loose context into a runnable agent graph. Use for: author a graph, runGraph, orchestrate agents as code, turn this workflow into agents, multi-agent pipeline with an audit trail." +--- + +# Codemode: ideas into agent graphs + +You are turning intent into a program, not into a conversation. +A graph is plain data — profiles as nodes, versioned prompt directives as edges — executed by `runGraph`, which composes `supervise()` and records every directive that crosses an edge in a ledger. +The output of this skill is never "an agent did things"; it is a graph whose ledger and deliverable verdict prove what happened. + +## Choose the dialect first + +| Problem shape | Use | Why | +| --- | --- | --- | +| Topology known before running; needs repeatability, caps, an audit trail | `runGraph` static graph (this skill) | Every traversal ledgered; caps fail loud; topology is reviewable data | +| Topology discovered while working (fan-out over a list you find mid-run) | `dynamic-workflows` skill — script over `spawn_agent`/`parallel` | A static graph cannot add nodes at runtime | +| One-shot-able by a single strong agent; no parallelism, no independent verification | No orchestration at all | Measured: a harness-driven root costs ~11× a plain router loop; composition must earn that | + +When unsure, write the deliverable check first (below). If one agent could produce bytes that pass it, you do not need a graph. + +## The contract, compressed + +`AgentGraph = { nodes, edges, deliverable, budget }` — all four mandatory; `runGraph(graph, options)` validates everything before any compute, so configuration faults throw instead of burning budget. + +**Nodes** — `{ id, profile }`. The profile is the entire description of the node: `prompt.systemPrompt` is its standing role, tools/mcp/resources its capabilities. `profile.name` must equal `id` — the name IS the node's identity for pinning and analyst routing. + +**Edges** — two kinds, both carrying a versioned directive (`PromptHandle`, e.g. `promptHandle('delegates/worker-brief/v1')`): + +- `delegates` `{ from, to, directive, maxTraversals? }` — work flowing from the one root to a worker. One edge per worker: to change the brief, register a new directive **version**, never a second edge. Each spawn *and* each mid-run steer consumes a traversal (default cap 32); the cap is the cyclic-run backstop and refuses loudly when exhausted. +- `analyzes` `{ analyst, over, to, directive, maxTraversals? }` — a lens (from `options.analysts`, **never a node id**) observing settled workers and routing findings to a node. Analysts are environment: if the rubric were a node, workers could address and game it. + +**Topology (current: P0)** — exactly one root (delegates, never delegated to); every delegates edge originates at the root; every other node must be reachable by a delegates edge. Deeper delegation chains are not yet expressible — do not fake them with prompt instructions. + +**Deliverable** — the independent termination oracle: `{ check: (out) => boolean, describe? }`. `describe` doubles as the root's task text — **the real mission goes here**, or your driver runs with a generic one-liner. `check` must accept genuinely-done output and reject junk; a throwing check is fail-closed (never delivered), so a shape mismatch silently burns the run to `budget-exhausted`. + +**Budget** — one conserved pool for the whole graph; cycles without conservation never terminate. + +## Budget from the floors, not from optimism + +`perWorker` defaults to a quarter of the pool. A `pi` worker spends **31,211 input tokens** (measured minimum, `WORKER_TOKEN_FLOOR`) before any useful work — so a pool under ~125k tokens with defaulted `perWorker` gets every spawn refused `below-runtime-floor`. Live history: five of six root-authored child budgets were below the floor before the refusal existed. + +- Workers on a measured harness: give each at least floor + working headroom (pi: ≥60k), and set `perWorker` explicitly. +- Workers on an unmeasured harness (`claude-code`, `codex`, …): the floor is **unknown, not zero** — budget generously and treat "settled with nothing produced" as a probable floor kill. +- A refused spawn still consumes an edge traversal. A driver retry-looping on refusals burns the cap; the refusal hint says to **raise** `maxTokens` — never retry smaller. + +## Authoring procedure + +1. **Deliverable first.** Write `check` as a mechanical test over bytes or state. If you cannot, you do not understand the task yet — stop and clarify, don't graph. +2. **Fewest distinct roles.** A node earns existence only if you would write its standing prompt differently from every other node's. Roles you cannot differentiate belong in one node. +3. **Author each node's profile**: `name` = id, `systemPrompt` = the standing role, capabilities only what that role needs. +4. **One delegates edge per worker** with a versioned brief; size `maxTraversals` as expected spawns + expected steers (they share the count, and only spawns are cap-checked). +5. **Analysts only for observation that must be unaddressable** — quality lenses, safety reads. `over` must list **worker** nodes: an `analyzes` over the root validates but never fires (the root never settles as a worker). +6. **Budget from step 4 of the floors section.** +7. **Run offline before live** (below), then swap the backend. + +## Prove it offline before spending + +Every topology is provable with zero network: inject a scripted `brain` for the driver, a stub `backend`/`makeWorkerAgent` for leaves, and default in-memory journal/blobs. The four `examples/graphs/` topologies are the templates. Offline the deliverable may be a rubber stamp (`out !== undefined`); live it must be real — a stamp makes the first settle "win" regardless of quality. + +## Read the evidence, or you shipped nothing + +A graph "worked" only if its records say so: + +| Read | For | +| --- | --- | +| `result.result.kind` + `reason` | winner / why not; `spentTotal` with `tokensKnown`/`usdKnown` — unknown is flagged, never zeroed | +| `result.ledger` | every traversal: `delivered` / `stripped` (authority narrowed it) / `empty` / `unpropagated`, with byte counts. An edge you expected with **zero traversals** = mis-routing | +| `result.exhaustedEdges` | **always** — `GraphEdgeCapError` throws only on a no-winner, non-lifecycle ending; a `budget-exhausted` run returns normally with exhausted caps listed | +| journal `edge` events | the durable twin of the ledger, for post-hoc and cross-run analysis | + +## Potholes + +| Pothole | Rule | +| --- | --- | +| `analyzes.over` includes the root | Validates, never fires. Lens over workers only | +| Analyzes `maxTraversals` used as a stop | It is observability-only; exhaustion ledgered `unpropagated`, never refuses. Only delegates caps stop a loop | +| Real task only in an imagined spawn payload | The driver's task is `deliverable.describe` — put the mission there | +| Second delegates edge to the same worker | Refused. Version the directive | +| Renaming a profile to "fix" a name/id mismatch | The equality is identity, not style — set both from one constant | +| Driver-authored spawn profiles adding capabilities | Ignored by design; only node selection crosses. Capabilities live in node profiles | +| Reading only for `GraphEdgeCapError` | Misses cap exhaustion on lifecycle endings — read `exhaustedEdges` | +| Trusting `$0` spend | `usdKnown: false` means unmetered, not free | + +## Improving this skill + +This file is an optimizable surface. `skills/codemode/IMPROVE.md` maps the loop — author-from-case → offline `runGraph` → deterministic ledger scoring → gated skill revision — onto agent-eval's existing machinery (`skillOptOptimizationMethod` takes skill text as its surface; `runImprovementLoop` gates promotion on a held-out set). Improve the skill through that loop, never by ad-hoc edits after a single bad run. + +## Then consider + +| Condition | Next skill | What to pass | +| --- | --- | --- | +| Worker profiles need real authoring | authoring-agent-profiles (supervisor-lab) | the node list and each role's one-line mission | +| Topology turns out runtime-discovered | dynamic-workflows (supervisor-lab) | the deliverable check and role prompts you already wrote | +| A skill revision needs its lift measured | generate-eval / eval-engineering | the cases and the ledger-scoring judge from IMPROVE.md | diff --git a/skills/codemode/cases/cap-as-stop-mistake.json b/skills/codemode/cases/cap-as-stop-mistake.json new file mode 100644 index 00000000..8fc6be4d --- /dev/null +++ b/skills/codemode/cases/cap-as-stop-mistake.json @@ -0,0 +1,8 @@ +{ + "id": "cap-as-stop-mistake", + "brief": "Have an analyst watch the worker and stop the whole thing after three findings.", + "expect": { + "trapIsAnalyzesCapAsStop": true, + "correctStopIsDelegatesCapOrDeliverable": true + } +} \ No newline at end of file diff --git a/skills/codemode/cases/floor-trap-pi.json b/skills/codemode/cases/floor-trap-pi.json new file mode 100644 index 00000000..caffd4f1 --- /dev/null +++ b/skills/codemode/cases/floor-trap-pi.json @@ -0,0 +1,11 @@ +{ + "id": "floor-trap-pi", + "brief": "Cheap little task: have a pi agent write a one-line file. Keep the budget tight, like a few thousand tokens.", + "expect": { + "nodes": 1, + "analyzesWarranted": false, + "floorTrap": true, + "mustBudgetAtLeast": 31211, + "correctAuthorOverridesBrief": true + } +} \ No newline at end of file diff --git a/skills/codemode/cases/mission-in-deliverable.json b/skills/codemode/cases/mission-in-deliverable.json new file mode 100644 index 00000000..d6a8640c --- /dev/null +++ b/skills/codemode/cases/mission-in-deliverable.json @@ -0,0 +1,8 @@ +{ + "id": "mission-in-deliverable", + "brief": "Produce a CHANGELOG entry for last week that passes our format check.", + "expect": { + "deliverableDescribeCarriesMission": true, + "checkIsMechanical": true + } +} \ No newline at end of file diff --git a/skills/codemode/cases/review-pipeline.json b/skills/codemode/cases/review-pipeline.json new file mode 100644 index 00000000..c5682f08 --- /dev/null +++ b/skills/codemode/cases/review-pipeline.json @@ -0,0 +1,14 @@ +{ + "id": "review-pipeline", + "brief": "I want code changes reviewed by two different perspectives before anything merges, and someone neutral deciding.", + "expect": { + "nodes": 3, + "analyzesWarranted": true, + "floorTrap": false, + "edges": [ + "delegates to each reviewer", + "analyzes routing findings to root" + ], + "wrongIfAnalystIsNode": true + } +} \ No newline at end of file diff --git a/skills/codemode/cases/runtime-discovered-fanout.json b/skills/codemode/cases/runtime-discovered-fanout.json new file mode 100644 index 00000000..5d612955 --- /dev/null +++ b/skills/codemode/cases/runtime-discovered-fanout.json @@ -0,0 +1,8 @@ +{ + "id": "runtime-discovered-fanout", + "brief": "Find every failing test in the repo and fix each one in parallel.", + "expect": { + "correctAnswerIsDynamicWorkflow": true, + "reason": "topology discovered mid-run" + } +} \ No newline at end of file diff --git a/skills/codemode/cases/single-agent-suffices.json b/skills/codemode/cases/single-agent-suffices.json new file mode 100644 index 00000000..c07c5a88 --- /dev/null +++ b/skills/codemode/cases/single-agent-suffices.json @@ -0,0 +1,7 @@ +{ + "id": "single-agent-suffices", + "brief": "Summarize this document into five bullets.", + "expect": { + "correctAnswerIsNoGraph": true + } +} \ No newline at end of file diff --git a/skills/codemode/cases/steer-heavy-drafting.json b/skills/codemode/cases/steer-heavy-drafting.json new file mode 100644 index 00000000..9ac9dc9a --- /dev/null +++ b/skills/codemode/cases/steer-heavy-drafting.json @@ -0,0 +1,9 @@ +{ + "id": "steer-heavy-drafting", + "brief": "One writer drafts, I want the coordinator to redirect it up to five times based on how the draft evolves.", + "expect": { + "nodes": 1, + "maxTraversalsAtLeast": 6, + "reason": "spawns and steers share the traversal count" + } +} \ No newline at end of file diff --git a/skills/codemode/cases/unmeasured-harness.json b/skills/codemode/cases/unmeasured-harness.json new file mode 100644 index 00000000..c68ec976 --- /dev/null +++ b/skills/codemode/cases/unmeasured-harness.json @@ -0,0 +1,9 @@ +{ + "id": "unmeasured-harness", + "brief": "Run three probes on claude-code workers and collect what they output.", + "expect": { + "nodes": 3, + "floorTrap": false, + "generousBudgetsBecauseFloorUnknown": true + } +} \ No newline at end of file From afb40bc1452427246fca077c61e59ed8f0371dec Mon Sep 17 00:00:00 2001 From: Drew Stone Date: Mon, 3 Aug 2026 11:28:52 -0600 Subject: [PATCH 2/2] fix(skills): codemode description within the 96-char gate --- skills/codemode/SKILL.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/skills/codemode/SKILL.md b/skills/codemode/SKILL.md index 2c543d35..05708970 100644 --- a/skills/codemode/SKILL.md +++ b/skills/codemode/SKILL.md @@ -1,6 +1,6 @@ --- name: codemode -description: "Convert an idea, workflow, or loose context into a runnable agent graph. Use for: author a graph, runGraph, orchestrate agents as code, turn this workflow into agents, multi-agent pipeline with an audit trail." +description: "Author agent graphs from ideas or loose context — runGraph, orchestration, workflows-as-code." --- # Codemode: ideas into agent graphs