Skip to content

Execute the serving cookbook's console transcripts, and mend their rot - #35

Merged
Shashankss1205 merged 1 commit into
mainfrom
fix/issue-2
Jul 31, 2026
Merged

Execute the serving cookbook's console transcripts, and mend their rot#35
Shashankss1205 merged 1 commit into
mainfrom
fix/issue-2

Conversation

@Shashankss1205

Copy link
Copy Markdown
Collaborator

What this does

docs/cookbook/06-serving-and-ops.md showed console transcripts that no test ever re-ran, and they had rotted exactly the way the issue predicted — and a little further:

  • /healthz showed "version":"0.1.0" while grapharc/__init__.py says 0.1.1
  • grapharc metrics has since grown events: and per_phase: fields (and nests per_node in --json) that the page did not show
  • replay trace.jsonl <id> | tail -4 was a line short (pending = [])

This PR copies the marker pattern from tests/test_cookbook_models.py (<!-- verified: cli --> / <!-- verified: cli varies --> / <!-- needs-credentials -->) into tests/test_cookbook_serving.py, marks every console block on the page, corrects the stale transcripts, and re-records the serve transcript against the current tree.

Byte-compared vs varying, and why

Byte-compared (verified: cli) — deterministic output:

  • the CLI tour (demo / trace / metrics / viz / replay / diff, 8 commands)
  • the --json pair (metrics … --json and the exit 2 failure document)
  • models openrouter/anthropic/claude-haiku-4.5 (pure spec resolution, no credentials)

These run through a real shell (bash -c, with a grapharc shim pinned to the test interpreter) as one session in one directory, because the later blocks read the trace file the tour wrote. Two spans the page itself declares variable are handled rather than skipped: the page's random run ids are remapped to the ids the re-run actually produced (in trace-file order), and wall-clock spans (duration_ms, (0.4ms) are masked. Every other character has to match — the demo block's leading ... elision compares the tail exactly.

Executed but not byte-compared (verified: cli varies) — genuinely machine-dependent:

  • the serve transcript: binds a port and mints random session/run ids. Replayed end-to-end against a real server — the page's own mygraphs.py is written to disk, the page's own command starts the server on a free port, and every curl in the block must succeed against it (the deterministic banner and /healthz body are still compared, and the /healthz version line is additionally pinned to grapharc.__version__ by name so it cannot show 0.1.0 against an 0.1.1 tree again).
  • models --check: probes this machine's credentials and PATH, and exits 1 when no real backend is usable, so only its shape is asserted — the same contract 02-models.md uses. This command was split out of the models <spec> block so the deterministic half stays byte-compared.

Never executed (needs-credentials):

  • grapharc agent … — always needs a live model; the test asserts it shows no output it never printed.

An unmarked console block now fails the suite, so a future edit cannot put an unverified transcript on the page.

Gate proof

Changed tokens: 81 to tokens: 82 in the CLI tour's expected output: test_console_command_prints_exactly_what_the_page_shows[line1595-step3] fails with a diff. Reverted.

Verification

  • uv run pytest tests/test_cookbook_serving.py — 57 passed
  • uv run pytest — 1628 passed; the only failures are 5 pre-existing SIGALRM/max_seconds timing tests that fail identically on untouched main on this machine and are unrelated to this change
  • uv run ruff check . — clean

Fixes #2

🤖 Generated with Claude Code

The five console transcripts in 06-serving-and-ops.md were the one place
the cookbook's guarantee did not reach: recorded once, never re-run, and
the page said so out loud. They had already rotted the way unexecuted
docs do — /healthz showed version 0.1.0 against an 0.1.1 tree,
`grapharc metrics` had grown `events` and `per_phase` fields the page
did not show, and `replay | tail -4` was a line short.

tests/test_cookbook_serving.py now marks and replays them the way
tests/test_cookbook_models.py does its page's. `verified: cli` — the CLI
tour, the `--json` pair and the `models <spec>` resolution — is re-run
through a real shell as one session and byte-compared, with the page's
random run ids mapped to the re-run's and wall-clock spans masked.
`verified: cli varies` — `serve`, replayed against a real server on a
free port with every curl required to succeed, and `models --check`,
which probes the host — runs for success but is not byte-compared, and
the /healthz version line it cannot compare is pinned to
grapharc.__version__ by name instead. `needs-credentials` — the `agent`
command, which always needs a live model — stays unexecuted and must
show no output. An unmarked transcript now fails the suite, and the
transcripts themselves were re-recorded against the current tree.

Fixes #2

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Shashankss1205
Shashankss1205 merged commit 37c6b41 into main Jul 31, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

docs: the console blocks in 06-serving-and-ops.md are never executed by a test

1 participant