[Harbor 4/4] architecture docs, tutorial, and the GAIA example

[varunursekar]

Draft · Stack 4 of 4 — targets harbor-3-compiler. Additive, low-risk.

• docs/harbor/architecture.md — what it is, the compiled-task topology, the two modes, the component map, and the leaderboard-integrity model.
• docs/harbor/tutorial.md — build + run end to end (both modes, the agent-side protocol); README Harbor section.
• examples/gaia-optimization — a Mode-B example optimizing a GaiaAgent (thin Terminus2 subclass with an editable prompt) on gaia/gaia via a nested harbor run on Modal.

Start your reading here for the big picture, then dive into [1/4]–[3/4].

Stack: [1/4] core → [2/4] sidecar → [3/4] compiler → this.

🤖 Generated with Claude Code

Greptile Summary

This stack-capping PR adds documentation and a runnable example for the Harbor integration: an architecture doc, an end-to-end tutorial, and the gaia-optimization Mode B example that optimizes a Terminus2-based GAIA agent's prompt via nested Harbor runs on Modal.

• docs/harbor/architecture.md and docs/harbor/tutorial.md document the compiled-task topology, the two evaluation modes, the leaderboard-integrity trust model (including an honest fail-open caveat for unlisted splits), and the full build + run workflow.
• examples/gaia-optimization ships a complete Mode B setup: a GaiaAgent (Terminus2 subclass with editable prompts), a build.yaml with explicit split access and a well-commented caveat about validation task-ID visibility through agent_repo, and a pyproject.toml that bundles the prompt files into the wheel.

Confidence Score: 5/5

This is an additive, docs-and-example-only PR with no changes to production code paths; safe to merge.

All changes are documentation files and a new example package. The only code is agent.py, a thin subclass that overrides prompt routing. No existing logic is modified. The two findings are editorial (typos in the upstream-copied prompt) and a defensive-coding suggestion (explicit fallthrough guard) — neither affects correctness of the current example.

No files require special attention. agent.py has a minor silent-fallthrough risk worth a quick read.

Important Files Changed

Filename	Overview
vero/README.md	Additive Harbor integration section with CLI quickstart and links to the new docs; no logic changes.
vero/docs/harbor/architecture.md	New architecture doc covering the compiled-task topology, two evaluation modes, component map, and leaderboard integrity; correctly documents the fail-open split default as a known limitation.
vero/docs/harbor/tutorial.md	New end-to-end tutorial with Mode A and Mode B build.yaml examples and the agent-side protocol; clear and accurate against the architecture.
vero/examples/gaia-optimization/README.md	Well-structured example README with prerequisites, run instructions, and an honest caveat about validation task-ID visibility through agent_repo.
vero/examples/gaia-optimization/build.yaml	Mode B build config with all splits explicitly listed (avoiding the fail-open default), a detailed caveat about held-out task-ID visibility, and correctly scoped secrets.
vero/examples/gaia-optimization/pyproject.toml	Standard hatchling src-layout config; force-include for the prompts directory ensures they are bundled into the wheel.
vero/examples/gaia-optimization/src/gaia_agent/init.py	Trivial re-export of GaiaAgent; correct.
vero/examples/gaia-optimization/src/gaia_agent/agent.py	Thin Terminus2 subclass redirecting prompt paths via _get_prompt_template_path; accesses private _parser_name — a silent fallthrough to the base-class path if the attribute's values drift upstream.
vero/examples/gaia-optimization/src/gaia_agent/prompts/terminus-json-plain.txt	Verbatim copy of the Harbor terminus-2 JSON prompt; no issues.
vero/examples/gaia-optimization/src/gaia_agent/prompts/terminus-xml-plain.txt	Copy of the Harbor terminus-2 XML prompt with two typos inherited from upstream: "apprpriate" and "In is always possible".

_{Reviews (3): Last reviewed commit: "docs(harbor): honest integrity guarantee..." | Re-trigger Greptile}

[shehabyasser-scale]

This reads as a hard guarantee ('the optimizer cannot read hidden labels, modify the scorer, or bypass its budget'), but the code makes each best-effort and the shipped GAIA example undercuts the first one (see the build.yaml comment). Suggest softening to something like: 'vero never writes per-sample labels to the agent's volume and meters every agent evaluation; OS-level mechanisms (read-only paths, a root:600 finalize token) keep the scorer and test split out of the agent's reach on a best-effort basis.'

[shehabyasser-scale]

'hidden; never reaches the optimizer' isn't true for this config: build.yaml is git-tracked and agent_repo is ., so vero harbor build seeds this whole file — including these validation task IDs — into /work/agent via git archive HEAD. The optimizer can read the held-out task IDs, and GAIA answers are public. Move the partition out of the agent_repo subtree, or caveat that for public benchmarks the held-out identity is visible (only per-sample scores are withheld). This is the example that backs the headline 'cannot read hidden labels' claim, so it is worth getting airtight.

[shehabyasser-scale]

examples/gsm8k-agent is cited as the Mode A example but it has no build.yaml (it's the older Policy-API example). The Harbor Mode A example that ships a build.yaml is examples/doubler-agent. Repoint here, or add a build.yaml to gsm8k-agent.

[shehabyasser-scale]

Worth one explicit line here: tier_for_split defaults any split not listed to viewable (full per-sample results), so omission fails open. Tell authors to list every split explicitly. (Pairs with the protocol.py fail-open comment on #4.)

[shehabyasser-scale]

'the scorer is sidecar-only' holds for Mode B but not Mode A, where the scorer lives in the agent's editable repo, protected only by chown root:root + chmod -R a-w on read_only_paths (which isn't a real tamper control — see #5). Recommend splitting this claim by mode.