Harness usage optimization

[Blankeos]

Summary

Harness usage optimization for crabcode’s agent loop — cut SuperGrok / API burn by aligning request shape, retention, and caps with OpenCode, Codex, and Grok Build.

Compaction stays manual (/compact) on purpose.

Ranked changes

Ordered by estimated impact on a typical multi-step coding session (tool-heavy, 20–100+ steps). Savings are relative to crabcode before this PR; ranges vary with session length and repo instruction size.

Rank	Change	Est. savings	Parity	What we did
1	Stop dumping tool JSON schemas into the system prompt	High — often thousands of prefix tokens × every step (schemas were paid twice)	OpenCode · Codex · Grok Build all send schemas only via API tools[]	System prompt now lists tool names + short guidance; full schemas stay on the provider tools field
2	Prune stale tool outputs in the live agent loop	High on long turns — often 10–40% of growing in-turn context	Grok Build soft-trim/hard-clear · OpenCode prune-to-placeholder	Keep newest 6 tool results full; soft-trim older large outputs (4k → 1.5k+1.5k); hard-clear older than 2× keep window to [Old tool result content cleared]; drop attached images on pruned outputs
3	Session-scoped prompt cache stickiness	Med–High — better prefix-cache hit rate (not fewer tokens sent, but cheaper / faster reuse)	Codex prompt_cache_key = session_id · Grok Build sticky session/cache keys · Anthropic ephemeral system cache	OpenAI Responses + OpenAI-compatible: prompt_cache_key = session_id; xAI: also store: false; Anthropic: cache_control: ephemeral on last system block
4	Tighten tool output caps	Med — large bash/read/grep steps shrink immediately	Grok Build bash ~20k / others 40k · OpenCode 50KiB · Codex ~10k tokens model-facing	Bash + terminal model output 20k (was 51k/60k); generic bridge 40k (was 60k)
5	Dedupe skills catalog	Med when skills are installed — catalog was resent in system + tool description every step	OpenCode: catalog in system only; skill tool desc is static	Skill tool description is short/static; <available_skills> stays in the system prompt
6	Lower default read limit	Med on read-heavy turns — ~30–50% less read payload when model uses defaults	Grok Build default 1000 lines · OpenCode 2000	Default read limit 1000 (was 2000); model can still pass limit/offset
7	Cap AGENTS.md / rule files at 32KiB	Med for fat instruction files — stable prefix every step	Codex project-doc max 32KiB · crabcode was 64KiB	DEFAULT_MAX_RULE_BYTES = 32 * 1024
8	Compact observation / question JSON	Low–Med	Avoid pretty-print bloat in model-facing text	Observation args use compact JSON + 4k cap (was pretty + 60k); question tool output compact

Intentionally not in this PR

Item	Why deferred
Auto-compaction	Manual /compact kept by design
xAI cli-chat-proxy.grok.com OAuth routing	Larger product/auth change; Grok Build’s blessed subscription gateway
Encrypted reasoning round-trip (ReasoningEncryptedContent)	Valuable for SuperGrok cache continuity; needs careful Responses item wiring — follow-up
Move “today’s date” out of system <env>	Cache-hygiene nicety; low immediate burn impact

Test plan

• Unit: responses_body_includes_prompt_cache_key_and_store_false
• Unit: tool output truncate / bash append caps
• Unit: soft_trim_tool_output_*, prune_stale_tool_outputs_clears_oldest_and_keeps_recent
• Manual: long SuperGrok/grok-composer-2.5-fast session — confirm usage % rises slower than before for similar work
• Manual: skills still load via skill tool using names from system <available_skills>

  

[cloudflare-workers-and-pages]

Deploying crabcode with    Cloudflare Pages

Latest commit:	5400518
Status:	✅  Deploy successful!
Preview URL:	https://3e765aa8.crabcode.pages.dev
Branch Preview URL:	https://cursor-xai-harness-efficienc.crabcode.pages.dev

View logs

[Blankeos]

PR Review: #6 — xAI / harness token-efficiency

Branch: cursor/xai-harness-efficiency-f44c → main
PR: #6
Title: Reduce agent token burn (prompt cache, tool-output prune, tighter caps)
Commits (3):

• 483a968 Reduce agent token burn: drop schema dump, add prompt cache keys, tighten tool caps
• 59dccc4 More harness usage wins: prune old tool outputs, dedupe skills, tighter defaults
• f13805e new bench (misnamed — only README streaming blurb)

Diff size: ~+445 / −107 across 20 files (mostly aisdk + prompt/tools)

---

Merge confidence: 4/5

Solid, well-scoped efficiency work with unit tests for the new prune/cache paths, clean compile, and no migrations. Docked one point for: (1) mid-stream prune mutates the in-loop message history that may feed session persistence / compaction, (2) subagents get prompt_cache_key: None, (3) stale __PARITY.md 2.5 claim, (4) f13805e is a noisy unrelated README edit under a misleading title.

CI (plan, Cloudflare Pages) is green; release jobs correctly skipped.

---

Body (GitHub comment paste)

## Summary
Cuts SuperGrok / long-session token burn by aligning the harness with OpenCode / Grok Build patterns:

1. **Stop double-sending tool schemas** — system prompt now lists tool *names* only; JSON schemas stay on the API `tools` field.
2. **OpenAI prompt cache** — stable `prompt_cache_key` (`crabcode:session:{id}`) + `store: false` for Responses API (primary + compaction paths).
3. **Mid-session tool-output prune** — keep last 6 full; soft-trim older; hard-clear beyond 12; drop attached images on pruned outputs.
4. **Dedupe skills** — skill catalog lives only in system prompt (`<available_skills>`), not re-embedded in the `skill` tool description every step.
5. **Tighter caps** — bash 20k, terminal model-facing 20k, bridge 30k, read default 1000 lines, rules 32KiB.

## Test plan
- [x] `cargo check -p crabcode` / `cargo check -p aisdk`
- [x] `cargo test -p aisdk prune_stale` / `soft_trim` / `responses_body_includes_prompt_cache_key`
- [x] `cargo test --bin crabcode truncate_tool_output`
- [ ] Manual multi-step agent turn (bash + read + skill) — confirm prune metadata + no tool-use regression
- [ ] Confirm OpenAI Responses request body has `prompt_cache_key` + `store: false`
- [ ] Long session: older tool outputs show `[Old tool result content cleared]` without breaking later steps

---

What changed (by area)

Area	Change
src/prompt/mod.rs	Drop pretty-printed JSON tool schemas from system prompt → name list only
src/tools/skill.rs	Remove embedded <available_skills> from tool description
src/aisdk/providers/openai.rs	prompt_cache_key + store override on Responses body
src/aisdk/providers/{compatible,anthropic}.rs	Builder plumbing only (no Anthropic ephemeral cache yet)
src/aisdk/response.rs	prune_stale_tool_outputs_in_place before each provider step
src/llm/client.rs + src/agent/config.rs	Wire session-stable cache key into primary + compaction LLM sessions
Tool caps	bash 51_200→20_000; terminal 60k→20k; bridge 100k→30k; read 2000→1000 lines; rules 64k→32k
README / npm README	Streaming bullet text tweak (f13805e)
_docs/__PARITY.md	Unrelated 2.3 wording tweak only

---

Regressions / risks

Medium

1. In-loop prune may poison persisted / returned history
   prune_stale_tool_outputs_in_place(&mut current_messages) runs on the live multi-step buffer in stream_with_tools. That buffer is what StreamTextResponse::messages() returns. If the app persists those messages into the session DB after a turn, old tool results are permanently replaced with [Old tool result content cleared] — fine for the next model step, bad for user scrollback / resume / compaction that expected full outputs.
   Check: does the session layer merge stream-final messages back into stored history, or only append new assistant/tool rows? If merge-from-stream, consider pruning a clone for the provider request only.

2. Subagents never get a cache key
   Subagent session build sets prompt_cache_key: None (src/llm/client.rs). Primary sessions benefit; task / explore / general subagents do not. Missed easy win, not a break.

3. Tighter caps can starve multi-file refactors

   • read default 1000 lines (was 2000) — large files need more offset/limit round-trips.
   • bash/terminal 20k — long test / build logs get truncated harder.
     Behavior is intentional but can look like “agent can’t see full output.”

Low

4. Models that ignore native tool schemas
   Prompt no longer carries full JSON schemas. Providers that only half-support function calling may degrade. Fine for OpenAI / Anthropic / xAI; worth watching on odd OpenAI-compatible endpoints.

5. force_store_false only on OpenAI Responses path
   Compatible chat-completions path accepts the option but does not emit equivalent fields (expected). Anthropic gets builder no-ops only — no cache_control: ephemeral yet (called out in PR body as follow-up).

6. _docs/__PARITY.md §2.5 is now wrong
   Still says: “src/tools/skill.rs repeats them in the tool description.” After this PR it does not. Should flip to “catalog only in system prompt.”

7. f13805e new bench is not a bench
   Only changes README streaming feature bullets. Misleading history; consider amending message or squashing before merge.

8. No Anthropic prompt-cache
   Token savings on Claude are prune/caps/schema only, not provider-side prefix cache.

Not regressions (good)

• Unit coverage for prune tiers, soft-trim, Responses body fields, bridge truncation.
• cargo check clean for crabcode + aisdk.
• PR checks: plan pass, Pages pass; release matrix correctly skipped.
• No auth / SQLite / models.dev cache format changes.

---

Migrations?

None.

Store	Touched?
SQLite prefs (~/.local/state/crabcode/data.db)	No schema / key changes
auth.json	No
models.dev cache JSON	No
Session transcript format	Behavioral only — if stream messages are written back, tool-output content may now be pruned placeholders (see risk #1). No version bump / migrator needed.

---

Pre-merge checklist

• CI plan green
• Related unit tests pass locally (aisdk prune/soft_trim/cache_key; crabcode truncate_tool_output)
• cargo check clean
• Decide on prune vs persistence — confirm session history is not rewritten with placeholders (or document that it is intentional)
• Fix __PARITY.md 2.5 (skill catalog location)
• Optional: set prompt_cache_key for subagent sessions (crabcode:subagent:{parent}:{task_id})
• Optional: rename/squash f13805e
• Smoke: multi-step BUILD turn on OpenAI Responses + one OpenAI-compatible (xAI) provider
• Smoke: skill tool still finds skills from system-prompt catalog only
• No just fmt / release-tag work required for this PR

---

Checks I ran (non-mutating)

Check	Result
gh pr view --json …	PR #6, base main, head cursor/xai-harness-efficiency-f44c
gh pr checks 6	plan pass, Cloudflare Pages pass; announce/build/publish skipping
cargo check -p crabcode	ok
cargo check -p aisdk	ok
cargo test -p aisdk prune_stale	pass
cargo test -p aisdk soft_trim	pass (2)
cargo test -p aisdk responses_body_includes_prompt_cache_key	pass
cargo test --bin crabcode truncate_tool_output	pass (2)

Note: local WIP on this branch (src/llm/xai_build.rs, dirty openai.rs / client.rs / mod.rs) was not part of this review — review is main...origin/cursor/xai-harness-efficiency-f44c only.

---

Verdict

Approve with nits — safe to merge after the persistence/prune sanity check and the one-line PARITY fix. Core idea is correct and tested; biggest unknown is whether pruned tool outputs leak into stored session history.

[cursor]

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f13805e1d5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Preserve xAI no-store behavior for agent mentions

This xAI-only store: false setup runs only on the primary stream_llm_with_cancellation path. The @agent flow in src/app.rs instead calls configure_subagent_llm_session, which builds its session through build_subagent_llm_session and leaves force_store_false at its default for API-key xAI users (OAuth happens to set it separately). Consequently, agent-mention subagent requests can be persisted server-side even though normal xAI turns use the new ZDR/no-store behavior; apply the same xAI option in the common subagent session setup.

Useful? React with 👍 / 👎.