From 82c6e3060d443f5b33726dbc58c93bbaff5390e7 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 3 Aug 2026 14:59:33 +0000 Subject: [PATCH 1/2] A bare backend name was read as a model name, and the CLI's failures said nothing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two defects that hid each other. Both found by installing 0.1.2 from PyPI into a clean venv and driving it against a real `claude` v2.1.220. **`split_spec` only consulted BACKENDS when the spec contained a slash.** A bare backend name fell through to "assume the default backend, keep the whole string as the model", so: split_spec("claude-cli") -> ("claude-cli", "claude-cli") split_spec("mock") -> ("claude-cli", "mock") `--model claude-cli` therefore shelled out to `claude -p --model claude-cli` and was refused by the CLI on every call (3/3), while `grapharc models --check` went on reporting the backend `usable` and `grapharc models claude-cli` printed `model: claude-cli` without complaint — two commands whose job is to say whether a spec will work, both saying yes about one that never did. `--model mock` was the worse half: it named the *paid* subscription backend and spawned the real binary, so the double `models --check` describes as "scripted test double; never reaches a provider" reached for one. It happened to fail before billing only because `mock` is not a model name. This is the same "silently folded into a model name … fails much later with a confusing error" failure `split_spec` already refuses for a mistyped backend *with* a slash; it just could not see the case without one. A bare backend name now resolves to that backend. `claude-cli` takes its own default model and `mock` takes the scripted double (which ignores the model segment entirely). `openrouter`, `openai` and `ollama` front catalogues rather than a model, so a bare name there is refused with a spelling that works rather than a guess about what to bill someone for. Slash forms and bare *model* names are untouched. **The gateway read the wrong stream.** `claude -p` fails with a non-zero exit, an empty stderr, and its whole explanation in the JSON envelope on stdout: {"is_error": true, "result": "There's an issue with the selected model (claude-cli). It may not exist or you may not have access to it."} `_invoke_cli` reported `proc.stderr`, so the error was `claude -p exited 1: ` — a sentence that stops at the colon. That is how the bug above presented itself: as no message at all. The one string that would have diagnosed it in seconds was captured, held in `proc.stdout`, and discarded. Note the code already parses this shape a few lines below, but only on the `returncode == 0` path, and the CLI sets `is_error` *and* exits non-zero, so it took the branch that ignores stdout. stdout is read first now, falling back to stderr when it is not the documented envelope. The recovered text also feeds `_cli_failure`, which classifies transient-vs-deterministic and was previously classifying from `""`. Tests: every new test was confirmed to fail against the old code and pass against the new — including the mock one, which asserts no subprocess is *created* rather than just checking the returned type, since the type was what was wrong and a refactor could fix the type and still shell out. `BARE_BACKEND_MODEL["claude-cli"]` is a second copy of the model class's default so the registry need not import a backend to split a string, and a test pins the two together the way CI already pins `__version__` to the packaged version. Verified: `grapharc demo stage1 --model claude-cli` now completes (8 nodes, target_met, exit 0) where it previously died with an empty error; `--model mock` resolves to the double with no subprocess; `--model openrouter` exits 2 with an example. Full suite green on 3.12 and 3.13; ruff clean. Co-Authored-By: Claude Opus 5 --- README.md | 3 +- grapharc/gateway/claude_cli.py | 34 +++++++- grapharc/gateway/registry.py | 45 ++++++++++ tests/test_gateway.py | 147 +++++++++++++++++++++++++++++++++ 4 files changed, 226 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 8f6360c..ed47f93 100644 --- a/README.md +++ b/README.md @@ -529,7 +529,8 @@ Re-derived on 2026-07-28 by running each item, not by reading the commit log. - **A planning round is an envelope, not a measurement.** A `round` event used to carry the planner's `tokens` and the round's `duration_ms`, both of which `metrics`, `cost` and `replay` add on top of node totals — and the planner's spend was already reported by its own `plan` event, so it was counted twice, and a round's duration encloses the plan plus every node it ran. Neither is on the event now; both are on its `state_delta` as `round_tokens` / `round_iterations` / `round_duration_ms`, where no reader sums them. `RoundRecord.iterations` also holds a figure now rather than always `0`. - **The Claude CLI backend is completion-only.** Tool calling and structured output need one of the OpenAI-wire backends: `openrouter`, `openai`, or a local `ollama`. - **A session turn is synchronous**, and a runner claim is a claim rather than a lease — nothing reclaims a session whose runner died holding it. -- **A bare model spec resolves to the paid `claude-cli` backend.** `--model mock` does not reach the scripted double; it becomes the model name `mock` on the subscription backend. Only the slash form (`mock/anything`) reaches the double. A mistyped backend *with* a slash is rejected properly, exit 2. +- *Closed:* a **bare backend name was read as a model name**, because `split_spec` only consulted the backend list when the spec contained a slash. `--model claude-cli` — the backend `models --check` reports as `usable` — shelled out to `claude -p --model claude-cli` and was refused by the CLI on *every* call, and `--model mock` named the paid subscription backend and spawned the real binary, so the double documented as "never reaches a provider" reached for one. A bare backend name now resolves to that backend (`claude-cli` to its own default model, `mock` to the scripted double, which ignores the model segment anyway); `openrouter`, `openai` and `ollama` front catalogues rather than a model, so those are refused with an example spelling instead of a guess about what to bill you for. The slash forms and bare *model* names are unchanged. +- *Closed:* a failing `claude -p` **reported no reason at all**. The CLI exits non-zero with an empty stderr and puts its explanation in the JSON envelope on stdout; the gateway read only stderr, so the error was `claude -p exited 1: ` — a sentence that stops at the colon. Since the wrong-model bug above presented itself exactly that way, the two hid each other. stdout is read first now, and the recovered text also feeds the transient-vs-deterministic classifier, which was previously deciding from `""`. - **`.env` is found by walking up parent directories; `grapharc.toml` is not.** The config layer refuses an upward search on purpose — a run must not be governed by a file you did not know about. The credential loader predates that decision and still searches upward, so the thing that *spends money* is discovered more eagerly than the thing that *constrains* it. - **`grapharc run` has no budget unless you give it one.** Set any of `--max-tokens`, `--max-iterations`, `--max-seconds`, or `--max-concurrency`; without them each dimension is unlimited and the gate admits a topology of any worst-case cost. diff --git a/grapharc/gateway/claude_cli.py b/grapharc/gateway/claude_cli.py index f9c0677..ba1fd57 100644 --- a/grapharc/gateway/claude_cli.py +++ b/grapharc/gateway/claude_cli.py @@ -121,6 +121,34 @@ def _canonical_model(model: str) -> str: return model.split("/", 1)[1] if model.startswith("anthropic/") else model +def _payload_error(stdout: str) -> str: + """The CLI's own explanation of a failure, which it writes to *stdout*. + + `claude -p` reports a refused model, an expired login and their siblings + with a non-zero exit code, an **empty stderr**, and the whole reason in the + JSON envelope on stdout:: + + {"is_error": true, "result": "There's an issue with the selected model + (claude-cli). It may not exist or you may not have access to it."} + + Reading only stderr therefore produced ``claude -p exited 1: `` — a + sentence that stops at the colon — and discarded the one string that said + what was wrong. It matters for classification too: `_cli_failure` decides + transient-vs-deterministic from this text, and it was deciding from "". + + Returns "" when stdout is not the documented envelope, so the caller falls + back to stderr rather than inventing a reason. + """ + try: + payload = json.loads(stdout) + except (TypeError, ValueError): # JSONDecodeError is a ValueError + return "" + if not isinstance(payload, dict): + return "" + result = payload.get("result") + return str(result).strip() if result else "" + + def _cli_failure(message: str, evidence: str) -> GatewayError: """Build the right error class for a CLI failure from its own text. @@ -219,8 +247,10 @@ def _invoke_cli(self, argv: list[str], prompt: str, cwd: str) -> dict[str, Any]: ) from exc if proc.returncode != 0: - stderr = proc.stderr.strip() - raise _cli_failure(f"claude -p exited {proc.returncode}: {stderr[:500]}", stderr) + # stdout first: the CLI puts its reason in the JSON envelope there + # and leaves stderr empty. See `_payload_error`. + detail = _payload_error(proc.stdout) or proc.stderr.strip() + raise _cli_failure(f"claude -p exited {proc.returncode}: {detail[:500]}", detail) try: payload = json.loads(proc.stdout) diff --git a/grapharc/gateway/registry.py b/grapharc/gateway/registry.py index cb3845f..a581568 100644 --- a/grapharc/gateway/registry.py +++ b/grapharc/gateway/registry.py @@ -49,6 +49,31 @@ class UnknownBackendError(Exception): "mock": "mock", } +#: What a *bare* backend name resolves to. A backend belongs here only when it +#: can pick a model without being told: `claude-cli` has a documented default, +#: and the `mock` double ignores the model segment entirely (`get_model` builds +#: a `ScriptedChatModel` with no model argument at all). The remote catalogues +#: are deliberately absent — `openrouter` alone fronts thousands of models and +#: no default would be defensible, so a bare `openrouter` is an error rather +#: than a guess about which model to bill you for. +#: +#: The `claude-cli` value is a second copy of `ClaudeCodeCLIChatModel.model`'s +#: own default, kept here so this module does not import a backend just to +#: split a string. `test_the_bare_claude_cli_default_matches_the_model_class` +#: fails if the two ever drift, the same way CI already pins `__version__` to +#: the packaged version. +BARE_BACKEND_MODEL = { + "claude-cli": "claude-sonnet-5", + "mock": "mock", +} + +#: A concrete spec to show someone who typed a bare backend that needs a model. +_BARE_BACKEND_EXAMPLE = { + "openrouter": "openrouter/anthropic/claude-sonnet-4.5", + "openai": "openai/gpt-4o-mini", + "ollama": "ollama/llama3.1", +} + def split_spec(spec: str) -> tuple[str, str]: """Split `backend/model` — bare names get the default backend. @@ -66,10 +91,30 @@ def split_spec(spec: str) -> tuple[str, str]: someone typing it means. Reaching the same model through the broker is still `openrouter/openai/gpt-4o-mini`, because only the first segment is ever read as a backend. + + A **bare backend name** is read as a backend too, which it was not before: + the slash test above meant `claude-cli` fell through to the last line and + became the *model* `claude-cli`, so `--model claude-cli` shelled out to + `claude -p --model claude-cli` and was refused by the CLI on every call — + while `models --check` went on reporting the backend `usable`. `--model + mock` was worse than useless: it named the paid subscription backend and + spawned the real binary, so the double documented as "never reaches a + provider" reached for one. That is the same "silently folded into a model + name … fails much later with a confusing error" failure this function + already refuses for a mistyped backend *with* a slash; it just could not + see the case without one. """ head, sep, rest = spec.partition("/") if sep and head in BACKENDS: return head, rest + if not sep and spec in BACKENDS: + if spec in BARE_BACKEND_MODEL: + return spec, BARE_BACKEND_MODEL[spec] + raise UnknownBackendError( + f"{spec!r} names a backend, not a model, and it has no default " + f"model to fall back on — write {spec}/, for example " + f"{_BARE_BACKEND_EXAMPLE[spec]!r}" + ) if sep and head not in KNOWN_AUTHORS: raise UnknownBackendError( f"unknown backend {head!r} in spec {spec!r}; expected one of: " diff --git a/tests/test_gateway.py b/tests/test_gateway.py index 1697f35..2f2d30b 100644 --- a/tests/test_gateway.py +++ b/tests/test_gateway.py @@ -81,3 +81,150 @@ def test_live_cli_completes_a_prompt(): assert "pong" in msg.content.lower() assert model.last_usage is not None assert model.last_usage["total_tokens"] > 0 + + +# ---- a bare backend name is a backend, not a model --------------------------- +# +# `split_spec` only consulted `BACKENDS` when the spec contained a slash, so a +# bare backend name fell through to "assume the default backend, keep the whole +# string as the model". Two live failures came out of that, and both are gates +# below rather than prose. + + +@pytest.mark.parametrize( + "spec, expected", + [ + ("claude-cli", ("claude-cli", "claude-sonnet-5")), + ("mock", ("mock", "mock")), + # The slash forms and the bare-model form must be untouched by the fix. + ("claude-cli/claude-sonnet-5", ("claude-cli", "claude-sonnet-5")), + ("mock/whatever", ("mock", "whatever")), + ("claude-sonnet-5", ("claude-cli", "claude-sonnet-5")), + ("anthropic/claude-haiku-4.5", ("claude-cli", "anthropic/claude-haiku-4.5")), + ], +) +def test_a_bare_backend_name_resolves_to_that_backend(spec, expected): + """`--model claude-cli` used to mean "the model called claude-cli". + + It shelled out to `claude -p --model claude-cli`, which the CLI refuses on + every call — `There's an issue with the selected model (claude-cli)` — + while `grapharc models --check` went on reporting the backend `usable` and + `grapharc models claude-cli` printed `model: claude-cli` without complaint. + Both commands exist to say whether a spec will work, and both said yes + about one that never did. + """ + from grapharc.gateway.registry import split_spec + + assert split_spec(spec) == expected + + +def test_the_mock_double_never_reaches_a_provider(): + """`grapharc models --check` calls mock a "scripted test double; never + reaches a provider". Bare `mock` used to resolve to the *paid* Claude CLI + backend and spawn the real binary — the one guarantee a double exists to + make, broken in the direction that costs money. + + Asserted on the behaviour (no subprocess is created), not just on the + returned type, because the type is what was wrong and a future refactor + could get the type right and still shell out. + """ + import subprocess + + from grapharc.gateway import get_model + from grapharc.testing import ScriptedChatModel + + spawned: list = [] + real_run, real_popen = subprocess.run, subprocess.Popen + subprocess.run = lambda *a, **k: spawned.append(a[0]) or real_run(*a, **k) + subprocess.Popen = lambda *a, **k: spawned.append(a[0]) or real_popen(*a, **k) + try: + model = get_model("mock", responses=["hi"]) + assert isinstance(model, ScriptedChatModel) + assert model.invoke("anything").content == "hi" + finally: + subprocess.run, subprocess.Popen = real_run, real_popen + + assert spawned == [], f"the mock double spawned a subprocess: {spawned}" + + +@pytest.mark.parametrize("backend", ["openrouter", "openai", "ollama"]) +def test_a_bare_backend_with_no_default_model_is_refused_with_an_example(backend): + """These front catalogues, not a model. Guessing one would be guessing what + to bill someone for, so the spec is refused — and the message has to carry + a spelling that works, because "that is wrong" without "this is right" is + what sent people to `--model openrouter` in the first place. + """ + from grapharc.gateway.registry import UnknownBackendError, split_spec + + with pytest.raises(UnknownBackendError) as caught: + split_spec(backend) + message = str(caught.value) + assert "names a backend, not a model" in message + assert f"{backend}/" in message + + +def test_the_bare_claude_cli_default_matches_the_model_class(): + """`BARE_BACKEND_MODEL` holds a second copy of the model class's own default + so the registry does not import a backend just to split a string. Two + declarations of one value drift; this is the check that they have not, in + the same spirit as CI pinning `__version__` to the packaged version. + """ + from grapharc.gateway.registry import BARE_BACKEND_MODEL + + assert ( + BARE_BACKEND_MODEL["claude-cli"] + == ClaudeCodeCLIChatModel.model_fields["model"].default + ) + + +# ---- the CLI's failures explain themselves ---------------------------------- + + +def test_a_failure_reports_the_reason_the_cli_wrote_to_stdout(): + """`claude -p` fails with a non-zero exit, an *empty stderr*, and the whole + reason in its JSON envelope on stdout. Reading only stderr produced + `claude -p exited 1: ` — a sentence that stops at the colon — and threw + away the one string that said what was wrong. That is how a wrong model + spec presented itself: as no message at all. + """ + import json + + from grapharc.gateway.claude_cli import _payload_error + + # The apostrophe in "There's" is the reason this is built with `json.dumps` + # and not an f-string with quotes swapped. + reason = "There's an issue with the selected model (claude-cli)." + stdout = json.dumps({"is_error": True, "result": reason}) + + assert _payload_error(stdout) == reason + # Not the envelope -> "" so the caller falls back to stderr rather than + # inventing a reason from whatever happened to be on stdout. + assert _payload_error("boom, not json") == "" + assert _payload_error("[1, 2]") == "" + assert _payload_error('{"is_error": true}') == "" + + +def test_a_failure_with_a_stdout_reason_beats_an_empty_stderr(monkeypatch): + """End to end through `_invoke_cli`, because the bug was in which stream it + read, not in parsing: the payload reader above can be perfect and the error + still be blank if the call site keeps reaching for `proc.stderr`. + """ + import subprocess + + from grapharc.gateway.errors import GatewayError + + reason = "There's an issue with the selected model (nope)." + + class _Proc: + returncode = 1 + stdout = __import__("json").dumps({"is_error": True, "result": reason}) + stderr = "" + + monkeypatch.setattr(subprocess, "run", lambda *a, **k: _Proc()) + model = ClaudeCodeCLIChatModel(model="nope") + + with pytest.raises(GatewayError) as caught: + model._invoke_cli(model._build_argv(None), "hi", ".") + + assert reason in str(caught.value) + assert not str(caught.value).endswith(": "), "the message stopped at the colon again" From 91d9551ce22b1c5bde7780488eeae8e1ec6cda42 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 3 Aug 2026 15:26:43 +0000 Subject: [PATCH 2/2] Restructure the README: one quick start, no duplicate claims, history in a CHANGELOG MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The prose was not the problem — it is dense, specific and honest, and it is lifted verbatim here. The shape around it was: **Two sections named some spelling of "quickstart".** `## Quick Start` (a Python graph) sat at the top and `## Quickstart` (the CLI tour) sat 100 lines below it, so the page opened on the smaller of two competing answers to the same question, and each linked to the other. The CLI tour is what actually demonstrates the project — eight demo stages, the governed planner, the observability commands, none of which need an API key — so it leads now, and the Python graph follows it as a single merged `## Quick start`. That also removes a third copy of the same idea: the CLI section already ended with its own "Building a graph" snippet nearly identical to the one at the top. **A bullet list restating a table, and restating itself.** "What makes GraphARC different" listed six claims already made — better, with mechanisms — by the "What it adds on top of LangGraph" table further down. Two of its six bullets ("Admission gate" and "Governed topology") were the same claim written twice. Dropped; the table is the version worth keeping. **Internals before installation.** The order was Quick Start, Architecture, Core Components, What it adds, Install, Quickstart — a reader met the module layout two sections before being told how to install anything. Now: Install, Quick start, the admission gate, what it adds, architecture, then subsystems. **A hand-maintained contents list in a different order from the document.** GitHub renders its own outline from the headings; a second one that had already drifted (it was missing eleven sections and, until recently, invented two) is a maintenance burden with no reader benefit. Removed. `Architecture` and `Core Components` were the diagram and a table of the same system, so they are one section. **Seven `*Closed:*` changelog entries filed under "Status and limits".** That made the list a mix of "still true" and "used to be true", which is the one distinction a reader of a limits section needs. They move to CHANGELOG.md in full — they are good engineering history and none of it is lost — and the section now says only what is still the case. Also fixed an inaccuracy the restructure surfaced: "the `run` stages use scripted models" described `grapharc demo`, not `grapharc run`, which is a different command. Tests. `test_the_quick_start_block_actually_runs_against_this_tree` asserted `== ["python"]` on the section's blocks, which encoded the old layout; it now selects the Python block by language, so the section can lead with bash and the snippet is still executed and compared against the result printed on the page. The contents-list link check became `test_every_in_page_link_resolves_to_a_real_heading` — it no longer depends on a section that no longer exists, and it covers every in-page link rather than only the ones in that list, which is strictly more than before. 546 -> 504 README lines with no prose deleted, only moved or de-duplicated. Full suite green on 3.12; ruff clean. Co-Authored-By: Claude Opus 5 --- CHANGELOG.md | 18 ++++ README.md | 247 ++++++++++++++++++------------------------- tests/test_readme.py | 41 ++++--- 3 files changed, 144 insertions(+), 162 deletions(-) create mode 100644 CHANGELOG.md diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..a3cad53 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,18 @@ +# Changelog + +Defects that have been closed, kept in full rather than summarised: what broke, +how it surfaced, and what the fix actually guarantees. These lived in the +README's *Status and limits* section, which made that list a mix of "still true" +and "used to be true" — the two things a reader most needs kept apart. + +Entries are newest-last within a release, matching the order they were written. + +## Unreleased + +- `grapharc plan` drives the governed loop; `PolicyEngine.edge_policy()` compiles the TOML document into the gate `AdmissionChecker` consults, and `grapharc plan --policy` is the caller; `grapharc demo --memory PATH` hands the shipped graphs the durable SQLite store. +- the shipped registry withheld the trace recorder from its `PlannerNode` and `Materializer`, so `grapharc plan` wrote a file with no `plan` event and **no `start`/`end` pair for any node it executed** — the paragraph above claiming otherwise was true of a hand-wired loop and false of the one the command drives. Both now get the recorder, and a test asserts the phase counts. +- a `max_seconds` past the platform's `time_t` — `float("inf")`, or a plausible "effectively unlimited" like `1e10` — used to **disable the deadline guard for the rest of the process**. `setitimer` raised *after* the SIGALRM handler was installed and the process-wide slot taken, leaking both, so every later run silently fell back to the mechanism that cannot unwind a blocking syscall: a 0.3s ceiling then took a 5s sleep to notice. Arming is undone on failure now, and the armed delay is clamped to what both mechanisms accept. +- every `async def` node **double-charged** its token re-reports. The re-report ledger was keyed by thread ident, but `on_llm_end` is sync — under `ainvoke` LangChain dispatches it to a worker thread while the body stays on the event loop — so the automatic charge found no ledger and the node's named re-report was charged again. Any node using the shipped `charge_usage`, `AgentNode._charge_tokens` or `planner.proposal._charge` reported double its real spend and hit `max_tokens` at half its declared allowance. The ledger is a `contextvars` scope now, which also fixes an inner scope discarding the enclosing node's. +- a bracket anywhere in a model's **prose** hijacked JSON extraction, because only the first `{`/`[` was ever tried. `Based on the context [lines 3-5]: {…}` was rejected as unparseable, and — worse — `Analysis (note [1]): {"supported": false}` returned a perfectly valid `[1]`, substituting a fabricated value for the verifier's actual answer. Every opener is tried now, and length alone turned out not to be a safe rank — a citation list like `[101, 205, 309, …]` *longer* than the verdict still won — so object spans are tried before array spans, each longest-first; junk still returns `None`, so fail-closed is unchanged. +- a **bare backend name was read as a model name**, because `split_spec` only consulted the backend list when the spec contained a slash. `--model claude-cli` — the backend `models --check` reports as `usable` — shelled out to `claude -p --model claude-cli` and was refused by the CLI on *every* call, and `--model mock` named the paid subscription backend and spawned the real binary, so the double documented as "never reaches a provider" reached for one. A bare backend name now resolves to that backend (`claude-cli` to its own default model, `mock` to the scripted double, which ignores the model segment anyway); `openrouter`, `openai` and `ollama` front catalogues rather than a model, so those are refused with an example spelling instead of a guess about what to bill you for. The slash forms and bare *model* names are unchanged. +- a failing `claude -p` **reported no reason at all**. The CLI exits non-zero with an empty stderr and puts its explanation in the JSON envelope on stdout; the gateway read only stderr, so the error was `claude -p exited 1: ` — a sentence that stops at the colon. Since the wrong-model bug above presented itself exactly that way, the two hid each other. stdout is read first now, and the recovered text also feeds the transient-vs-deterministic classifier, which was previously deciding from `""`. diff --git a/README.md b/README.md index ed47f93..8e50429 100644 --- a/README.md +++ b/README.md @@ -17,127 +17,14 @@ Build production-grade multi-agent systems with built-in safety, auditability, and control. GraphARC adds a governance layer on top of LangGraph: a planner *proposes* a subgraph, a deterministic checker *admits* it, and only then does anything execute. Every transition is permitted, every loop is bounded, and afterwards you can prove what happened and why it stopped. -**Status:** Early days (`0.1.2`) — the API is not stable yet. `pip install grapharc` — see [Install](#install). +**Status:** early days (`0.1.2`) — the API is not stable yet. Known limits are listed in [Status and limits](#status-and-limits); closed ones are in [CHANGELOG.md](CHANGELOG.md). ![One English question is decomposed by a local model into a nine-node graph — four parallel evidence pulls fanning out of START, a correlate join, a hypothesis fork, and a final report — shown live in the browser: the proposed graph waits grey for human approval, then each node turns amber while it runs and green when it is done.](docs/media/grapharc-decompose.gif) *One question in, a governed graph out: a local model proposes the topology, the admission gate and a human approval decide, and the live view shows every node run — amber while executing, green when done. ([full-quality mp4](docs/media/grapharc-decompose.mp4))* -### What makes GraphARC different - -- **Admission gate**: All runtime topology changes go through deterministic approval before execution -- **Typed state contracts**: Pydantic-based state validation at every node boundary -- **Per-node write permissions**: Explicit control over what each node can modify -- **Enforced budgets**: Token spend and execution time limits with hard cutoffs -- **Complete auditability**: JSONL traces that serve as replay points and proof of execution -- **Governed topology**: New nodes and edges proposed at runtime are checked before being built - > *Graph engineering*: when one agent loop stops being enough, coordination becomes the engineering. Nodes do work (agent loops, model calls, deterministic functions, humans approving things), edges decide what runs next, and a typed shared state flows between them. GraphARC implements the discipline that makes such graphs production-grade rather than demos — the ideas emerging from the July 2026 loops-vs-graphs debate (Steinberger, Ng, et al.), the "Two Graphs, Two Jobs" split, and twenty years of pre-AI graph systems where every edge means something and every path can be explained. -## Table of Contents - -- [Quick Start](#quick-start) -- [Architecture](#architecture) -- [Core Components](#core-components) -- [What it adds on top of LangGraph](#what-it-adds-on-top-of-langgraph) -- [Install](#install) -- [Quickstart](#quickstart) — the CLI tour -- [The admission gate](#the-admission-gate) -- [Configuration, and the zero-config path](#configuration-and-the-zero-config-path) -- [The model gateway](#the-model-gateway) -- [Independent verification](#independent-verification) -- [Tools and the harness](#tools-and-the-harness) -- [Memory](#memory) -- [Sessions, the HTTP API, and policy](#sessions-the-http-api-and-policy) -- [Reading a run afterwards](#reading-a-run-afterwards) -- [Tests are gates](#tests-are-gates) -- [Status and limits](#status-and-limits) - -## Quick Start - -```python -from grapharc import Budget, GraphARC, GraphARCState -from grapharc.runtime.graph import END, START - -# State is a typed contract, not a free-form dict: `extra="forbid"`. -class MyState(GraphARCState): - question: str - result: str = "" - -def process(state: MyState) -> dict: - return {"result": f"handled: {state.question}"} - -graph = GraphARC(MyState, name="quickstart", budget=Budget(max_iterations=10)) - -# `writes` is required. A node that returns a field it did not declare -# raises WritePermissionError instead of quietly writing it. -graph.add_node("process", process, writes={"result"}) -graph.add_edge(START, "process") -graph.add_edge("process", END) - -print(graph.compile().invoke({"question": "hello"})) -# {'question': 'hello', 'result': 'handled: hello'} -``` - -Three things in that snippet are the whole point, and none of them are optional: -the state is a typed schema, the node declares what it may write, and the run -carries a `Budget`. See [Quickstart](#quickstart) for the CLI tour and -[The admission gate](#the-admission-gate) for the part with no prior art. - -## Architecture - -![The GraphARC architecture: a CLI or HTTP request reaches a planner, which emits a typed proposal; a deterministic admission checker either refuses it with reasons or admits it; only an admitted proposal is materialised and run by the graph kernel, on top of the model, tool and memory planes; everything lands on one JSONL record, and work discovered mid-run re-enters the gate.](docs/diagrams/architecture.png) - -The amber curve along the top is the claim: refusals return as traced reason codes, and work discovered mid-run **re-enters admission** — there is no already-approved path and no cached authorisation. - -For detailed architecture views, see [`docs/diagrams/grapharc-architecture.drawio`](docs/diagrams/grapharc-architecture.drawio) and [five more views](docs/diagrams/) generated from [`architecture.py`](docs/diagrams/architecture.py). - -## Core Components - -| Component | Purpose | Module | -|---|---|---| -| **Kernel** | Typed state contracts, declared writes, budgets, traces, fan-out, async support | `grapharc.runtime` | -| **Planner + Admission** | Propose subgraphs, admit/reject with reasons, materialise, replan | `grapharc.planner` | -| **Agent Node** | Observe → model → permission check → sandboxed tool → repeat loop | `grapharc.harness` | -| **Tools** | Seven core tools with workspace confinement; container executor | `grapharc.tools` | -| **Sessions** | Long-lived, resumable across processes, human approval gates | `grapharc.session` | -| **HTTP API** | FastAPI + Server-Sent Events for streaming | `grapharc.server` | -| **Policy** | TOML rules over nodes, edges, tools and spend; decision audit trail | `grapharc.policy` | -| **Memory** | Durable claims with provenance, artifacts, BM25F + graph retrieval | `grapharc.memory` | -| **Observability** | Replay, run diffing, OpenTelemetry spans, cost attribution | `grapharc.observe` | - -Every component above is reachable from a shipped command. See [ROADMAP.md](ROADMAP.md) §12 for known gaps — the HTTP API still runs its own in-process session layer instead of the durable one. - -## What it adds on top of LangGraph - -Everything in this table is enforced by the library rather than left to convention, and has a test you can run. - -| Discipline | Mechanism | -|---|---| -| Write permissions | Every node declares which state fields it may write. An undeclared write raises `WritePermissionError`; plain LangGraph applies it and moves on. | -| State isolation | Nodes receive `state.model_copy(deep=True)`, so mutating a nested model in place cannot sneak past the declared write channel — the returned dict is the only way out of a node. | -| Typed state | Schemas are Pydantic models with `extra="forbid"`. Declaring a write to a field that doesn't exist fails when the node is added, not when it runs. | -| Earned cycles | `dag=True` rejects conditional and fan-out edges when they're added, and cycles at compile time — Stage 0 before Stage 2. | -| Code-only routing | Routers are ordinary Python functions over typed state, so model prose cannot steer an edge. A node may also return `Command(goto=…)` for dynamic routing — still code — and the destination is validated against the compiled graph at the node boundary. | -| Convergence | `ProgressGuard` returns the first triggered `StopReason` (target met / no progress / round cap), so a cycle ends with a machine-readable reason instead of running out of road. | -| Traces | Each node execution writes JSONL `start` / `end` / `error` events. `start` carries only the identity of the step (run, thread, attempt, graph, node, step, timestamp); `end` adds the state delta, duration and tokens; `error` adds the duration and the exception. `metrics`, `viz`, `replay`, `diff` and the OTel exporter read that same file, so the dashboard and the audit trail cannot disagree. | -| Fail-closed entry points | Driving a compiled graph through raw LangGraph (`.inner.invoke()`) raises `MissingRunContextError` rather than silently running with no budget and no trace. | -| Bounded work | A per-run `Budget` (iterations / tokens / seconds / concurrency). Iterations and tokens are metered by the runtime itself, and `max_seconds` is delivered as an interrupt *into* the running node rather than only checked between them. | -| Checked state edits | `update_state()` is not a passthrough: it rejects unknown fields, type-checks the values, and — given `as_node=` — applies that node's declared write allowlist. | -| Governed topology | New nodes and edges proposed at runtime go through a deterministic admission gate before anything is built. See [The admission gate](#the-admission-gate). | - -Three of those need their edges stated, because the gap is where people get hurt. - -**Budgets.** Tokens are charged without the node's cooperation: a LangChain callback is installed for the duration of every node, so any chat model invoked on that thread reports usage to the run's meter — including calls buried inside library code the node merely calls — and the ceiling is enforced at the node boundary. `max_seconds` is an interrupt, not a poll: SIGALRM on the main thread, an asynchronous exception otherwise, so a node parked in `time.sleep` or on a provider's socket is cut off at the deadline. Where it stops short: spend a provider never reports cannot be charged, a model invoked on a thread the node started itself is outside the callback's context, and an async exception cannot unwind a thread sitting inside a C call — it lands when that call returns. Even then the deadline holds at the node boundary: a node that overran does not get its writes into state. - -**Routing.** The routers are code, which is the property that matters: no model output is ever consulted to pick an edge. But `add_conditional_edge` passes the router and its mapping straight through to LangGraph — GraphARC does not verify that the router's return value is a key in the mapping, so a typo surfaces as a `KeyError` at run time rather than when the edge is added. - -**Typing.** Writes are checked in both directions: the dict a node returns is validated field by field against the state schema before it lands, and the state is validated again when the next node receives it. A value that doesn't fit raises `StateTypeError` naming the node, the field, the declared type and what arrived — and that includes the last node before `END`, so a bad type no longer escapes into the result. The validated value is what gets written, so a schema that says `int` means the result holds an `int`. The remaining gap is narrow and worth stating exactly: write-time validation is built from each field's *annotation*, so constraints carried in the annotation (`Annotated[int, Field(gt=0)]`) do bite, but a validator the state model declares for itself — `@field_validator`, `@model_validator` — is not run on a write. A node returning `{"slug": "NOT-LOWER"}` into a field whose validator demands lowercase is accepted, even though constructing the model directly with that value raises; the violation surfaces only when a later node receives the state and the whole model is rebuilt, which means one written by the last node before `END` still reaches the result. The write *allowlist* is GraphARC's; the *types* are Pydantic's. - -**Crash-safe resume is LangGraph's**, not GraphARC's: a checkpointer handed to `compile()` goes straight to `StateGraph.compile()`. What GraphARC adds on top is trace continuity — after a resume, step numbers continue from the thread's history and the attempt counter increments, so replay points stay unique across attempts. What `grapharc.session` adds on top of *that* is everything the kernel deliberately does not know about: who is driving the thread, what has been said to it since it last ran, and whether a human still has to sign something off. - -**Async is carried through.** `ainvoke`, `astream` and `astream_events` all run through the same disciplined path — budgets, traces and write permissions apply unchanged — and `async def` nodes execute. The sync entry points refuse a graph containing them with `AsyncNodeError` *before* anything runs, rather than letting LangGraph execute every sync node first and fail at the first coroutine. `astream_events` offers `v1` and `v2`; `v3` is refused because LangGraph returns a stream object there rather than an async iterator, which is a different contract than the method's. - ## Install Python >= 3.12. @@ -147,7 +34,7 @@ pip install grapharc # or: uv pip install grapharc grapharc demo stage0 # costs nothing, needs no key ``` -That is the whole install. The bare package carries the kernel, the planner and the admission gate, the agent harness, the seven core tools, memory, policy and the observability commands — everything the Quickstart below runs. Backends and the HTTP API are extras, because each pulls dependencies you should not pay for unless you use them: +That is the whole install. The bare package carries the kernel, the planner and the admission gate, the agent harness, the seven core tools, memory, policy and the observability commands — everything the quick start below runs. Backends and the HTTP API are extras, because each pulls dependencies you should not pay for unless you use them: ```bash pip install 'grapharc[openrouter]' # tool calling, structured output @@ -170,7 +57,11 @@ uv sync --group dev # Python >= 3.12 uv sync --all-extras --group dev # everything, plus the dev group ``` -## Quickstart + +## Quick start + +Nothing below needs an API key or a paid account: the demo stages run on +scripted models, and the observability commands read a file. ```bash grapharc demo stage0 # deterministic DAG: load -> split -> count -> report @@ -204,7 +95,7 @@ Twelve commands, and every one of them takes `--json` — in JSON mode the failu The Slack bot puts most of these commands one `/grapharc …` away from a phone, behind an allowlisting gate that keeps the default spend at zero — setup in [docs/cookbook/07-slack.md](docs/cookbook/07-slack.md), and a command-by-command session, refusals included, in [docs/cookbook/08-slack-walkthrough.md](docs/cookbook/08-slack-walkthrough.md). A tracing command run from Slack is narrated live — one status message edited in place as nodes run, with a refreshed diagram link — and `grapharc serve --live-root` adds a browser page that redraws the orchestration graph in real time over SSE. -The `run` stages use scripted models by default, so they cost nothing and produce the same trace every time. Add `--model` to run one against a real backend — that works for stage1 through stage6 and the capstone; stage0 is pure code with no model in it. `grapharc agent` is the exception: it needs a tool-calling backend and says so rather than degrading, because a scripted model has no `bind_tools` to drive a tool loop with. +The `demo` stages use scripted models by default, so they cost nothing and produce the same trace every time. Add `--model` to run one against a real backend — that works for stage1 through stage6 and the capstone; stage0 is pure code with no model in it. `grapharc agent` is the exception: it needs a tool-calling backend and says so rather than degrading, because a scripted model has no `bind_tools` to drive a tool loop with. Building a graph: @@ -224,8 +115,14 @@ g.add_node("answer", answer, writes={"answer"}) # undeclared writes raise g.add_edge(START, "answer") g.add_edge("answer", END) print(g.compile().invoke({"question": "meaning of life"})) +# {'question': 'meaning of life', 'answer': '42 (asked: meaning of life)'} ``` +Three things in that snippet are the whole point, and none are optional: the +state is a typed schema, the node declares what it may write, and the run +carries a `Budget`. + + ## The admission gate The part with no prior art to copy, and the reason the rest exists. You cannot pre-author a graph for "investigate this incident" — the shape is discovered while working. So the graph is built at runtime, and a deterministic checker stands between building it and running it. @@ -349,25 +246,60 @@ Three limits, because this is exactly the sort of claim people over-read: **One surface, one demo registry.** `grapharc plan` drives the loop, and the kinds it plans over come from `grapharc/examples/plan_incident.py` unless you point `--registry module:attr` at your own. That module is also where a custom registry declares its `STATE_SCHEMA` and `WRITES`; a registry alone is not enough, because a kind nobody declared writes for may write nothing. There is no session-backed or HTTP-backed planning surface yet — [ROADMAP.md](ROADMAP.md) §12.3. -## Configuration, and the zero-config path -Three flags carry every run: `--registry` (what may be proposed), `--policy` (what may connect to what), `--model`. Typing them repeatedly is how people stop using a tool, so they can come from a file: +## What it adds on top of LangGraph -```toml -# grapharc.toml -[grapharc] -registry = "myco.incident:build_registry" -policy = "policy.toml" -max_rounds = 6 -``` +Everything in this table is enforced by the library rather than left to convention, and has a test you can run. -Resolution is `flag > env (GRAPHARC_*) > grapharc.toml > built-in`, and **every value reports which layer supplied it** — `--json` carries a `sources` block, the human view prints a `config` line. A config file makes "which policy was I subject to" *less* visible on the command line, so the provenance is part of the output rather than something a reader reconstructs. +| Discipline | Mechanism | +|---|---| +| Write permissions | Every node declares which state fields it may write. An undeclared write raises `WritePermissionError`; plain LangGraph applies it and moves on. | +| State isolation | Nodes receive `state.model_copy(deep=True)`, so mutating a nested model in place cannot sneak past the declared write channel — the returned dict is the only way out of a node. | +| Typed state | Schemas are Pydantic models with `extra="forbid"`. Declaring a write to a field that doesn't exist fails when the node is added, not when it runs. | +| Earned cycles | `dag=True` rejects conditional and fan-out edges when they're added, and cycles at compile time — Stage 0 before Stage 2. | +| Code-only routing | Routers are ordinary Python functions over typed state, so model prose cannot steer an edge. A node may also return `Command(goto=…)` for dynamic routing — still code — and the destination is validated against the compiled graph at the node boundary. | +| Convergence | `ProgressGuard` returns the first triggered `StopReason` (target met / no progress / round cap), so a cycle ends with a machine-readable reason instead of running out of road. | +| Traces | Each node execution writes JSONL `start` / `end` / `error` events. `start` carries only the identity of the step (run, thread, attempt, graph, node, step, timestamp); `end` adds the state delta, duration and tokens; `error` adds the duration and the exception. `metrics`, `viz`, `replay`, `diff` and the OTel exporter read that same file, so the dashboard and the audit trail cannot disagree. | +| Fail-closed entry points | Driving a compiled graph through raw LangGraph (`.inner.invoke()`) raises `MissingRunContextError` rather than silently running with no budget and no trace. | +| Bounded work | A per-run `Budget` (iterations / tokens / seconds / concurrency). Iterations and tokens are metered by the runtime itself, and `max_seconds` is delivered as an interrupt *into* the running node rather than only checked between them. | +| Checked state edits | `update_state()` is not a passthrough: it rejects unknown fields, type-checks the values, and — given `as_node=` — applies that node's declared write allowlist. | +| Governed topology | New nodes and edges proposed at runtime go through a deterministic admission gate before anything is built. See [The admission gate](#the-admission-gate). | -**It does not search parent directories.** git, npm and cargo all walk upward; this deliberately doesn't. A run must never be silently governed by a policy file in a directory you didn't know about. Read from the working directory, or name one with `--config PATH`. A relative path *inside* a config resolves against the config, so the file means the same thing from anywhere. +Three of those need their edges stated, because the gap is where people get hurt. -**With nothing configured at all**, a run still works. [`grapharc.stdlib`](grapharc/stdlib.py) ships general-purpose node kinds — `collect_context`, `investigate`, `verify`, `summarize`, and `apply_change`, which is registered *and denied by default*. No phase anywhere is given `run_command`. If a model is available and no policy was named, one is **generated**, written to `.grapharc/generated-policy.toml` with a `REVIEW THIS` header, and reported as `policy_source: generated`. The second run reads it off disk as an ordinary file — so generation is a one-time state, and promoting it to a policy you own is an edit and a `mv`. +**Budgets.** Tokens are charged without the node's cooperation: a LangChain callback is installed for the duration of every node, so any chat model invoked on that thread reports usage to the run's meter — including calls buried inside library code the node merely calls — and the ceiling is enforced at the node boundary. `max_seconds` is an interrupt, not a poll: SIGALRM on the main thread, an asynchronous exception otherwise, so a node parked in `time.sleep` or on a provider's socket is cut off at the deadline. Where it stops short: spend a provider never reports cannot be charged, a model invoked on a thread the node started itself is outside the callback's context, and an async exception cannot unwind a thread sitting inside a C call — it lands when that call returns. Even then the deadline holds at the node boundary: a node that overran does not get its writes into state. + +**Routing.** The routers are code, which is the property that matters: no model output is ever consulted to pick an edge. But `add_conditional_edge` passes the router and its mapping straight through to LangGraph — GraphARC does not verify that the router's return value is a key in the mapping, so a typo surfaces as a `KeyError` at run time rather than when the edge is added. + +**Typing.** Writes are checked in both directions: the dict a node returns is validated field by field against the state schema before it lands, and the state is validated again when the next node receives it. A value that doesn't fit raises `StateTypeError` naming the node, the field, the declared type and what arrived — and that includes the last node before `END`, so a bad type no longer escapes into the result. The validated value is what gets written, so a schema that says `int` means the result holds an `int`. The remaining gap is narrow and worth stating exactly: write-time validation is built from each field's *annotation*, so constraints carried in the annotation (`Annotated[int, Field(gt=0)]`) do bite, but a validator the state model declares for itself — `@field_validator`, `@model_validator` — is not run on a write. A node returning `{"slug": "NOT-LOWER"}` into a field whose validator demands lowercase is accepted, even though constructing the model directly with that value raises; the violation surfaces only when a later node receives the state and the whole model is rebuilt, which means one written by the last node before `END` still reaches the result. The write *allowlist* is GraphARC's; the *types* are Pydantic's. + +**Crash-safe resume is LangGraph's**, not GraphARC's: a checkpointer handed to `compile()` goes straight to `StateGraph.compile()`. What GraphARC adds on top is trace continuity — after a resume, step numbers continue from the thread's history and the attempt counter increments, so replay points stay unique across attempts. What `grapharc.session` adds on top of *that* is everything the kernel deliberately does not know about: who is driving the thread, what has been said to it since it last ran, and whether a human still has to sign something off. + +**Async is carried through.** `ainvoke`, `astream` and `astream_events` all run through the same disciplined path — budgets, traces and write permissions apply unchanged — and `async def` nodes execute. The sync entry points refuse a graph containing them with `AsyncNodeError` *before* anything runs, rather than letting LangGraph execute every sync node first and fail at the first coroutine. `astream_events` offers `v1` and `v2`; `v3` is refused because LangGraph returns a stream object there rather than an async iterator, which is a different contract than the method's. + + +## Architecture + +![The GraphARC architecture: a CLI or HTTP request reaches a planner, which emits a typed proposal; a deterministic admission checker either refuses it with reasons or admits it; only an admitted proposal is materialised and run by the graph kernel, on top of the model, tool and memory planes; everything lands on one JSONL record, and work discovered mid-run re-enters the gate.](docs/diagrams/architecture.png) + +The amber curve along the top is the claim: refusals return as traced reason codes, and work discovered mid-run **re-enters admission** — there is no already-approved path and no cached authorisation. + +For detailed architecture views, see [`docs/diagrams/grapharc-architecture.drawio`](docs/diagrams/grapharc-architecture.drawio) and [five more views](docs/diagrams/) generated from [`architecture.py`](docs/diagrams/architecture.py). + +| Component | Purpose | Module | +|---|---|---| +| **Kernel** | Typed state contracts, declared writes, budgets, traces, fan-out, async support | `grapharc.runtime` | +| **Planner + Admission** | Propose subgraphs, admit/reject with reasons, materialise, replan | `grapharc.planner` | +| **Agent Node** | Observe → model → permission check → sandboxed tool → repeat loop | `grapharc.harness` | +| **Tools** | Seven core tools with workspace confinement; container executor | `grapharc.tools` | +| **Sessions** | Long-lived, resumable across processes, human approval gates | `grapharc.session` | +| **HTTP API** | FastAPI + Server-Sent Events for streaming | `grapharc.server` | +| **Policy** | TOML rules over nodes, edges, tools and spend; decision audit trail | `grapharc.policy` | +| **Memory** | Durable claims with provenance, artifacts, BM25F + graph retrieval | `grapharc.memory` | +| **Observability** | Replay, run diffing, OpenTelemetry spans, cost attribution | `grapharc.observe` | + +Every component above is reachable from a shipped command. See [ROADMAP.md](ROADMAP.md) §12 for known gaps — the HTTP API still runs its own in-process session layer instead of the durable one. -**Policy is generated; a registry never is.** Policy is data, and the worst case is bad rules you can read. A registry holds *functions*, so generating one would mean a model writing code that then executes — and the gate would be checking a list the gated thing wrote. The model selects from the shipped kinds instead. Selecting is safe; authoring is not. ## The model gateway @@ -406,14 +338,6 @@ OpenRouter also carries routing: model-level `fallback_models` chains, provider Caveats each backend accepts openly. **Claude CLI:** `bind_tools` and `with_structured_output` raise `NotImplementedError` — the adapter implements neither, so what you get is LangChain's `BaseChatModel` default, and `claude -p` in the tool-free mode GraphARC drives it in offers nothing to implement them with. GraphARC also has no cache control on this path — the CLI decides and the usage envelope reports what it did — and calls spend subscription quota. **OpenRouter:** credit is reserved against `max_tokens`, so the default is deliberately modest. **OpenAI:** the API returns token counts and no price, so `cost_usd` is `None` and a dollar ceiling counts calls instead of enforcing — pass `price_per_million=` or price the trace afterwards with `observe.cost.RateCard`; token budgets are unaffected. **Ollama:** free by definition, so calls are charged `0.0` rather than counted as unpriced, and whether tool-calling works depends on the model you pulled rather than on the adapter. **All of them:** the per-call `cost_usd` is captured, budgeted against, *and* written onto the trace, so `observe.cost` reports a recorded figure rather than an estimate whenever the provider gave one. See [ROADMAP.md](ROADMAP.md) §10.4 for what is still missing (a tenant on the event). -## Independent verification - -`verify_claim` is the piece worth copying even if you use none of the rest. - -- **The anchor runs before the model.** The citation must appear verbatim in the source (whitespace is the only latitude, so a paraphrase is still caught). A fabricated quote is rejected with the reviewer's call count still at zero — a hallucinated citation costs nothing. -- **The reviewer gets a fresh context.** If the anchor holds, the reviewer sees only the claim, the quote, and a mechanically extracted window of surrounding source — never the author's conversation. That window is what lets it catch a real quote lifted out of a negated sentence. -- **Ambiguity fails closed.** An unparseable reply, a non-boolean `supported` value, or a citation under 12 characters is a rejection. -- **Independence is enforced in two places, both worth knowing exactly.** `build_stage5` and `build_capstone` refuse the *same object* for author and reviewer — an identity check, which will not catch two separate instances of the same model. The CLI does the stronger check: `different_providers()` compares the *vendor* each spec reaches — the model author when the id names one, the backend's own vendor otherwise, so a Claude-CLI author and an Anthropic model over OpenRouter are correctly read as correlated — and `grapharc demo --model … --reviewer-model …` warns when the pair shares a vendor, because correlated agreement is exactly what the verifier exists to prevent. ## Tools and the harness @@ -433,6 +357,7 @@ Seven core tools — `read_file`, `write_file`, `edit_file`, `list_dir`, `glob`, **Where a real boundary is needed, `ContainerExecutor` is it.** Same `run(spec, args)` interface, so callers never branch on which executor they hold. It runs each tool call in a throwaway container with one bind mount (the workspace), `--network none` unless the tool declared `needs_network`, all capabilities dropped, `no-new-privileges`, a non-root uid, a read-only rootfs, and memory and pid limits. Its constraints are enforced rather than documented away: the tool must be resolvable *inside the image* — a lambda, a `functools.partial` or a bound method is refused before a container starts, and a derived import path that cannot be checked without running host code is refused *there* instead, contained — and arguments and results must survive JSON. The image is part of the security decision and is yours to choose; the default `python:3.12-slim` contains no GraphARC and none of your code, so running your own tools means building an image that has them. + ## Memory Claims carry provenance — source, observation time, and the run that produced them — and corrections are recorded by supersession rather than overwrite, so a later run can see that a fact was replaced and skip the dead end. Entity resolution is Unicode-aware, so 東京 and 北京 stay distinct instead of both collapsing to an empty key. @@ -447,6 +372,7 @@ Claims carry provenance — source, observation time, and the run that produced The limit to know: **the in-process store is still the default.** `grapharc demo stage6` and `grapharc demo capstone` keep claims in a dict for the life of the process unless you pass `--memory PATH`, which hands them the `SQLiteMemoryStore` — verified to survive across two separate interpreters, not just two calls in one. In-process stays the default so a plain run writes nothing you did not ask for. The durable stores that exist are `SQLiteMemoryStore` (no extra needed) and `LadybugMemoryStore` (the `ladybug` extra); there is no extra beyond those. + ## Sessions, the HTTP API, and policy **Sessions survive a process restart.** A `SessionManager` over a directory keeps status, the event queue, approval holds and the audit trail in SQLite, with graph state in the kernel's checkpointer. Verified by running it: one interpreter created a session, ran two nodes and stopped `awaiting_approval` holding a gated node; a second interpreter resumed it by id, saw the hold, approved it, and ran the rest — with each node appearing exactly once in an append-only log, so nothing was repeated and nothing skipped. The resuming process must register the graph in its own registry, or it gets `UnknownGraphError` rather than a guess. @@ -457,6 +383,17 @@ Three things that phrase over-promises if left alone. **An interrupt does not st **Policy is a TOML document** over nodes, edges, tools and spend, with tiered evaluation — every `deny` before every `ask` before every `allow`, so a broad deny beats a narrow allow including one scoped to a single tenant. Every decision lands in an audit record naming the rule id, the reason, the policy version and a digest of the document, so a decision can be tied to the exact text that made it. And the seam, now narrowed to exactly half: **the edge half is wired and the tool half is not.** `PolicyEngine.edge_policy()` compiles the document into the `EdgePolicy` the admission checker consults, and `grapharc plan --policy` is a real caller — so what may connect to what *is* governed by a document you can read. But `permission_policy()`, `check_tool()` and `approval_router()` have no caller outside `grapharc/policy/`, so `grapharc agent` still assembles its tool gating from `--allow` / `--deny` / `--ask` globs. The most dangerous surface in the package is the one the document cannot reach yet; [issue #6](https://github.com/CodeGraphContext/GraphARC/issues/6) is that work, and the precedence question it has to settle is what happens when a flag `allow` meets a document `deny`. + +## Independent verification + +`verify_claim` is the piece worth copying even if you use none of the rest. + +- **The anchor runs before the model.** The citation must appear verbatim in the source (whitespace is the only latitude, so a paraphrase is still caught). A fabricated quote is rejected with the reviewer's call count still at zero — a hallucinated citation costs nothing. +- **The reviewer gets a fresh context.** If the anchor holds, the reviewer sees only the claim, the quote, and a mechanically extracted window of surrounding source — never the author's conversation. That window is what lets it catch a real quote lifted out of a negated sentence. +- **Ambiguity fails closed.** An unparseable reply, a non-boolean `supported` value, or a citation under 12 characters is a rejection. +- **Independence is enforced in two places, both worth knowing exactly.** `build_stage5` and `build_capstone` refuse the *same object* for author and reviewer — an identity check, which will not catch two separate instances of the same model. The CLI does the stronger check: `different_providers()` compares the *vendor* each spec reaches — the model author when the id names one, the backend's own vendor otherwise, so a Claude-CLI author and an Anthropic model over OpenRouter are correctly read as correlated — and `grapharc demo --model … --reviewer-model …` warns when the pair shares a vendor, because correlated agreement is exactly what the verifier exists to prevent. + + ## Reading a run afterwards `trace` is the only writer. `metrics`, `replay`, `diff`, `cost` and the OTel exporter all read that one file and nothing else, which is what keeps a dashboard from contradicting an audit trail. @@ -473,6 +410,28 @@ grapharc metrics trace.jsonl # tokens, retries, termination rea **Cost attribution is per run, thread and node**, and it distinguishes what was measured from what was guessed. Tokens are counted from the same events `metrics` uses — node `end` events plus work that happened outside any node span, which is what a `grapharc agent` run is entirely made of — and the suite asserts the two agree, so a cost report and an audit trail cannot drift apart. The provider's own `cost_usd` is written onto the trace, so `recorded_cost_usd` holds a real figure when the backend reported one; a backend that reports none falls back to tokens priced against a `RateCard` you supply, and the two never mix. There is still no tenant on a trace event, so per-tenant attribution is not offered rather than being approximated. + +## Configuration, and the zero-config path + +Three flags carry every run: `--registry` (what may be proposed), `--policy` (what may connect to what), `--model`. Typing them repeatedly is how people stop using a tool, so they can come from a file: + +```toml +# grapharc.toml +[grapharc] +registry = "myco.incident:build_registry" +policy = "policy.toml" +max_rounds = 6 +``` + +Resolution is `flag > env (GRAPHARC_*) > grapharc.toml > built-in`, and **every value reports which layer supplied it** — `--json` carries a `sources` block, the human view prints a `config` line. A config file makes "which policy was I subject to" *less* visible on the command line, so the provenance is part of the output rather than something a reader reconstructs. + +**It does not search parent directories.** git, npm and cargo all walk upward; this deliberately doesn't. A run must never be silently governed by a policy file in a directory you didn't know about. Read from the working directory, or name one with `--config PATH`. A relative path *inside* a config resolves against the config, so the file means the same thing from anywhere. + +**With nothing configured at all**, a run still works. [`grapharc.stdlib`](grapharc/stdlib.py) ships general-purpose node kinds — `collect_context`, `investigate`, `verify`, `summarize`, and `apply_change`, which is registered *and denied by default*. No phase anywhere is given `run_command`. If a model is available and no policy was named, one is **generated**, written to `.grapharc/generated-policy.toml` with a `REVIEW THIS` header, and reported as `policy_source: generated`. The second run reads it off disk as an ordinary file — so generation is a one-time state, and promoting it to a policy you own is an edit and a `mv`. + +**Policy is generated; a registry never is.** Policy is data, and the worst case is bad rules you can read. A registry holds *functions*, so generating one would mean a model writing code that then executes — and the gate would be checking a list the gated thing wrote. The model selects from the shipped kinds instead. Selecting is safe; authoring is not. + + ## Tests are gates Each stage ships a failure-gate test, not just a happy path: @@ -498,6 +457,7 @@ uv run pytest -m live # real backends: spends money and quota Live tests are deselected by default via `addopts` in `pyproject.toml`, so a plain `pytest` never reaches a real model — verified: a plain run reports 10 deselected. `--strict-markers` is on, and a misspelled marker is a collection error rather than a test that silently spends money. + ## Status and limits Re-derived on 2026-07-28 by running each item, not by reading the commit log. @@ -511,17 +471,12 @@ Re-derived on 2026-07-28 by running each item, not by reading the commit log. **Built and unreachable** — this used to be the honest headline, four subsystems deep. One seam is left. - **The HTTP API does not use the durable session layer.** It has its own `InProcessRuntime`, whose sessions die with the process and whose approvals are recorded without being delivered. [ROADMAP.md](ROADMAP.md) §12.3. -- *Closed:* `grapharc plan` drives the governed loop; `PolicyEngine.edge_policy()` compiles the TOML document into the gate `AdmissionChecker` consults, and `grapharc plan --policy` is the caller; `grapharc demo --memory PATH` hands the shipped graphs the durable SQLite store. -- *Closed:* the shipped registry withheld the trace recorder from its `PlannerNode` and `Materializer`, so `grapharc plan` wrote a file with no `plan` event and **no `start`/`end` pair for any node it executed** — the paragraph above claiming otherwise was true of a hand-wired loop and false of the one the command drives. Both now get the recorder, and a test asserts the phase counts. **Real limits of things that do work** - **Admission authorises a kind, not its arguments.** A proposal carrying `args={"path": "/etc/passwd"}` is admitted on the strength of its kind alone. - **The audit-hook sandbox is in-process confinement, not a kernel boundary.** `os.stat` outside the workspace is not blocked, because CPython raises no event for it. `ContainerExecutor` is the boundary where one is needed. - **`run_command` is not confined.** Argv-only and never a shell, but the child is an ordinary process with your privileges. -- *Closed:* a `max_seconds` past the platform's `time_t` — `float("inf")`, or a plausible "effectively unlimited" like `1e10` — used to **disable the deadline guard for the rest of the process**. `setitimer` raised *after* the SIGALRM handler was installed and the process-wide slot taken, leaking both, so every later run silently fell back to the mechanism that cannot unwind a blocking syscall: a 0.3s ceiling then took a 5s sleep to notice. Arming is undone on failure now, and the armed delay is clamped to what both mechanisms accept. -- *Closed:* every `async def` node **double-charged** its token re-reports. The re-report ledger was keyed by thread ident, but `on_llm_end` is sync — under `ainvoke` LangChain dispatches it to a worker thread while the body stays on the event loop — so the automatic charge found no ledger and the node's named re-report was charged again. Any node using the shipped `charge_usage`, `AgentNode._charge_tokens` or `planner.proposal._charge` reported double its real spend and hit `max_tokens` at half its declared allowance. The ledger is a `contextvars` scope now, which also fixes an inner scope discarding the enclosing node's. -- *Closed:* a bracket anywhere in a model's **prose** hijacked JSON extraction, because only the first `{`/`[` was ever tried. `Based on the context [lines 3-5]: {…}` was rejected as unparseable, and — worse — `Analysis (note [1]): {"supported": false}` returned a perfectly valid `[1]`, substituting a fabricated value for the verifier's actual answer. Every opener is tried now, and length alone turned out not to be a safe rank — a citation list like `[101, 205, 309, …]` *longer* than the verdict still won — so object spans are tried before array spans, each longest-first; junk still returns `None`, so fail-closed is unchanged. - **`interrupt()` suspends but cannot be resumed.** LangGraph's native interrupt stops the graph and shows on `get_state`, and there is no supported resume path — resuming means passing a `Command` as *input*, which is closed by design. Use the session layer's approval gate for human-in-the-loop. - **Still unwrapped from LangGraph:** `retry_policy`, `cache_policy`, `durability`, subgraphs. `.inner` reaches them, but execution entry points there fail closed, so `.inner` is an inspection escape hatch and not a way to run the graph. - **Cost is recorded when a backend reports one, estimated when it does not.** Both gateways publish the provider's `cost_usd` through the same `llm_output` envelope, the runtime's usage callback writes it onto the node's `end` event, and an agent's `model` events carry the per-call breakdown. A backend that reports no price still falls back to a `RateCard` estimate, and the two figures stay apart — `recorded_cost_usd` is never a guess. Still missing: no tenant on a trace event, so per-tenant attribution is not offered. @@ -529,8 +484,6 @@ Re-derived on 2026-07-28 by running each item, not by reading the commit log. - **A planning round is an envelope, not a measurement.** A `round` event used to carry the planner's `tokens` and the round's `duration_ms`, both of which `metrics`, `cost` and `replay` add on top of node totals — and the planner's spend was already reported by its own `plan` event, so it was counted twice, and a round's duration encloses the plan plus every node it ran. Neither is on the event now; both are on its `state_delta` as `round_tokens` / `round_iterations` / `round_duration_ms`, where no reader sums them. `RoundRecord.iterations` also holds a figure now rather than always `0`. - **The Claude CLI backend is completion-only.** Tool calling and structured output need one of the OpenAI-wire backends: `openrouter`, `openai`, or a local `ollama`. - **A session turn is synchronous**, and a runner claim is a claim rather than a lease — nothing reclaims a session whose runner died holding it. -- *Closed:* a **bare backend name was read as a model name**, because `split_spec` only consulted the backend list when the spec contained a slash. `--model claude-cli` — the backend `models --check` reports as `usable` — shelled out to `claude -p --model claude-cli` and was refused by the CLI on *every* call, and `--model mock` named the paid subscription backend and spawned the real binary, so the double documented as "never reaches a provider" reached for one. A bare backend name now resolves to that backend (`claude-cli` to its own default model, `mock` to the scripted double, which ignores the model segment anyway); `openrouter`, `openai` and `ollama` front catalogues rather than a model, so those are refused with an example spelling instead of a guess about what to bill you for. The slash forms and bare *model* names are unchanged. -- *Closed:* a failing `claude -p` **reported no reason at all**. The CLI exits non-zero with an empty stderr and puts its explanation in the JSON envelope on stdout; the gateway read only stderr, so the error was `claude -p exited 1: ` — a sentence that stops at the colon. Since the wrong-model bug above presented itself exactly that way, the two hid each other. stdout is read first now, and the recovered text also feeds the transient-vs-deterministic classifier, which was previously deciding from `""`. - **`.env` is found by walking up parent directories; `grapharc.toml` is not.** The config layer refuses an upward search on purpose — a run must not be governed by a file you did not know about. The credential loader predates that decision and still searches upward, so the thing that *spends money* is discovered more eagerly than the thing that *constrains* it. - **`grapharc run` has no budget unless you give it one.** Set any of `--max-tokens`, `--max-iterations`, `--max-seconds`, or `--max-concurrency`; without them each dimension is unlimited and the gate admits a topology of any worst-case cost. @@ -538,10 +491,14 @@ Re-derived on 2026-07-28 by running each item, not by reading the commit log. [ROADMAP.md](ROADMAP.md) tracks what is built and what is not, item by item. +Defects that have been **closed** — each with what broke, how it was found and what the fix actually guarantees — are in [CHANGELOG.md](CHANGELOG.md). They were moved there because a list headed *Status and limits* should say what is still true. + + ## Design lineage Architecturally *inspired by* systems studied from public documentation: OpenClaw (policy-before-schema tool gating, file-first state, and its security post-mortems), Hermes Agent (budgeted tiered memory, ephemeral subagents), Claude Code (advisory-vs-enforced split, subagent context isolation, verification-centered loops), and OpenRouter (routing semantics, budget-scoped accounting). + ## License MIT — see [LICENSE](LICENSE). diff --git a/tests/test_readme.py b/tests/test_readme.py index 0e41fd3..cf41e32 100644 --- a/tests/test_readme.py +++ b/tests/test_readme.py @@ -104,16 +104,21 @@ def test_the_quick_start_block_actually_runs_against_this_tree(): trailing comment on the page states the printed result, and it is compared against what the block really prints. """ - blocks = _blocks("Quick Start") - assert [lang for lang, _ in blocks] == ["python"], blocks - code = blocks[0][1] + # The section leads with the CLI tour (a bash block) and follows it with one + # Python graph, so this selects by language rather than by position — the + # earlier `== ["python"]` assertion encoded the old layout, in which the + # Python snippet was the section's only content and sat 100 lines above the + # commands that actually demonstrate the project. + python_blocks = [code for lang, code in _blocks("Quick start") if lang == "python"] + assert len(python_blocks) == 1, "the quick start must carry exactly one Python block" + code = python_blocks[0] # The expectation is written on the page as a trailing `# {...}` comment, # so the snippet stays copy-pasteable instead of carrying a second block. expected = [ line.lstrip("# ").strip() for line in code.splitlines() if line.startswith("# {") ] - assert len(expected) == 1, "the Quick Start must state its printed result" + assert len(expected) == 1, "the quick start must state its printed result" buffer = io.StringIO() namespace: dict = {"__name__": "__readme__"} @@ -123,25 +128,27 @@ def test_the_quick_start_block_actually_runs_against_this_tree(): assert _normalise(buffer.getvalue()) == _normalise(expected[0]) -def test_the_quick_start_reaches_no_live_backend(): +def test_the_quick_start_python_block_reaches_no_live_backend(): """The first snippet a visitor copies must not be able to spend money.""" - code = _blocks("Quick Start")[0][1] + code = [c for lang, c in _blocks("Quick start") if lang == "python"][0] for forbidden in ("get_model(", "openrouter", "ClaudeCodeCLIChatModel", "claude-cli"): assert forbidden not in code, forbidden -def test_every_table_of_contents_link_resolves_to_a_real_heading(): - """Two of the seven entries pointed at sections that do not exist. +def test_every_in_page_link_resolves_to_a_real_heading(): + """A dead `#anchor` does nothing visible on GitHub, so neither reading the + page nor running the suite used to surface one. - `#usage` and `#documentation` were both dead — the refactor that added the - contents list invented them — and a dead anchor on GitHub silently does - nothing when clicked, so neither reading the page nor running the suite - surfaced it. Anchors are derived here the way GitHub derives them, and - every in-page link in the list has to land somewhere. + This began life checking only the hand-maintained contents list, which had + two invented entries (`#usage`, `#documentation`) pointing at sections that + never existed. That list is gone — GitHub renders its own outline, and a + second one in a different order from the document was a maintenance burden + that had already drifted — so the check now covers *every* in-page link on + the page, which is strictly more than it covered before and no longer + depends on a particular section existing. """ text = README.read_text(encoding="utf-8") - contents = _section("Table of Contents") anchors = set() for line in text.splitlines(): @@ -151,11 +158,11 @@ def test_every_table_of_contents_link_resolves_to_a_real_heading(): slug = re.sub(r"[^a-z0-9\s-]", "", title.lower()) anchors.add(re.sub(r"\s+", "-", slug.strip())) - linked = re.findall(r"\]\(#([a-z0-9-]+)\)", contents) - assert linked, "the contents list has no in-page links at all" + linked = re.findall(r"\]\(#([a-z0-9-]+)\)", text) + assert linked, "the README has no in-page links at all" dead = sorted(set(linked) - anchors) - assert not dead, f"table of contents links to non-existent sections: {dead}" + assert not dead, f"README links to non-existent sections: {dead}" def _embedded_images() -> list[str]: