Execute the serving cookbook's console transcripts, and mend their rot - #35
Merged
Conversation
The five console transcripts in 06-serving-and-ops.md were the one place the cookbook's guarantee did not reach: recorded once, never re-run, and the page said so out loud. They had already rotted the way unexecuted docs do — /healthz showed version 0.1.0 against an 0.1.1 tree, `grapharc metrics` had grown `events` and `per_phase` fields the page did not show, and `replay | tail -4` was a line short. tests/test_cookbook_serving.py now marks and replays them the way tests/test_cookbook_models.py does its page's. `verified: cli` — the CLI tour, the `--json` pair and the `models <spec>` resolution — is re-run through a real shell as one session and byte-compared, with the page's random run ids mapped to the re-run's and wall-clock spans masked. `verified: cli varies` — `serve`, replayed against a real server on a free port with every curl required to succeed, and `models --check`, which probes the host — runs for success but is not byte-compared, and the /healthz version line it cannot compare is pinned to grapharc.__version__ by name instead. `needs-credentials` — the `agent` command, which always needs a live model — stays unexecuted and must show no output. An unmarked transcript now fails the suite, and the transcripts themselves were re-recorded against the current tree. Fixes #2 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
docs/cookbook/06-serving-and-ops.mdshowed console transcripts that no test ever re-ran, and they had rotted exactly the way the issue predicted — and a little further:/healthzshowed"version":"0.1.0"whilegrapharc/__init__.pysays0.1.1grapharc metricshas since grownevents:andper_phase:fields (and nestsper_nodein--json) that the page did not showreplay trace.jsonl <id> | tail -4was a line short (pending = [])This PR copies the marker pattern from
tests/test_cookbook_models.py(<!-- verified: cli -->/<!-- verified: cli varies -->/<!-- needs-credentials -->) intotests/test_cookbook_serving.py, marks every console block on the page, corrects the stale transcripts, and re-records the serve transcript against the current tree.Byte-compared vs varying, and why
Byte-compared (
verified: cli) — deterministic output:demo/trace/metrics/viz/replay/diff, 8 commands)--jsonpair (metrics … --jsonand theexit 2failure document)models openrouter/anthropic/claude-haiku-4.5(pure spec resolution, no credentials)These run through a real shell (
bash -c, with agrapharcshim pinned to the test interpreter) as one session in one directory, because the later blocks read the trace file the tour wrote. Two spans the page itself declares variable are handled rather than skipped: the page's random run ids are remapped to the ids the re-run actually produced (in trace-file order), and wall-clock spans (duration_ms,(0.4ms) are masked. Every other character has to match — the demo block's leading...elision compares the tail exactly.Executed but not byte-compared (
verified: cli varies) — genuinely machine-dependent:servetranscript: binds a port and mints random session/run ids. Replayed end-to-end against a real server — the page's ownmygraphs.pyis written to disk, the page's own command starts the server on a free port, and everycurlin the block must succeed against it (the deterministic banner and/healthzbody are still compared, and the/healthzversion line is additionally pinned tographarc.__version__by name so it cannot show 0.1.0 against an 0.1.1 tree again).models --check: probes this machine's credentials and PATH, and exits 1 when no real backend is usable, so only its shape is asserted — the same contract02-models.mduses. This command was split out of themodels <spec>block so the deterministic half stays byte-compared.Never executed (
needs-credentials):grapharc agent …— always needs a live model; the test asserts it shows no output it never printed.An unmarked console block now fails the suite, so a future edit cannot put an unverified transcript on the page.
Gate proof
Changed
tokens: 81totokens: 82in the CLI tour's expected output:test_console_command_prints_exactly_what_the_page_shows[line1595-step3]fails with a diff. Reverted.Verification
uv run pytest tests/test_cookbook_serving.py— 57 passeduv run pytest— 1628 passed; the only failures are 5 pre-existing SIGALRM/max_secondstiming tests that fail identically on untouchedmainon this machine and are unrelated to this changeuv run ruff check .— cleanFixes #2
🤖 Generated with Claude Code