Harness usage optimization

.devrefs/devrefs.yml

@@ -13,3 +13,7 @@ references:
     description: "a ratatui agent harness that we can use as reference, especially since it's by openai"
     remote: https://github.com/openai/codex.git
     branch: main
+  - id: xai-org/grok-build
+    description: "source code for xai grok"
+    remote: https://github.com/xai-org/grok-build.git
+    branch: main

README.md

@@ -23,7 +23,7 @@ A purely Rust-based AI CLI coding agent with a beautiful terminal UI for interac
   - **Multiple Model Support** - Works w/ the same models.dev support.
   - **Command System** - Intuitive commands: `/sessions`, `/new`, `/connect`, `/models`, `/exit` + custom commands.
   - **Session Management** - Create and manage multiple chat sessions
-  - **Streaming Responses** - Real-time streaming of AI responses (w/ [aisdk.rs](https://aisdk.rs))
+  - **Streaming Responses** - Real-time streaming of AI responses + websockets using OpenAI.
 
 ## Installation
 
@@ -118,9 +118,9 @@ Like any benchmark, please take this with a grain of salt. I have a cherry-picke
 
 | Agent       | Score | Checks | Avg time | Est. tokens | Est. cost |
 | ----------- | ----: | -----: | -------: | ----------: | --------: |
-| 🦀 crabcode |  100% |  19/19 |    21.3s |        2851 |   $0.0103 |
-| 🔲 opencode |  100% |  19/19 |    24.2s |        3837 |   $0.0201 |
-| ⚛️ codex    |  100% |  19/19 |    24.9s |       56172 |   $0.5435 |
+| 🦀 crabcode |  100% |  19/19 |    29.8s |        2768 |   $0.0094 |
+| 🔲 opencode |  100% |  19/19 |    34.9s |        4612 |   $0.0279 |
+| ⚛️ codex    |  100% |  19/19 |    33.7s |       36888 |   $0.3506 |
 
 ## Contributing
 

_docs/__PARITY.md

@@ -25,7 +25,7 @@ Recent implementation notes:
 | 1.7  | Configurable max steps per agent                   | Per-agent `max_steps` controls agent loop depth.                                                 | **Present.** Agent registry supports `steps`, `maxSteps`, and `max_steps`; app/print/task paths read max steps from registry definitions.                                                 | No material max-step config gap beyond the duplicated runner logic noted in 1.3.                                                                                                                                           |
 | 2.1  | Provider-specific header and behavior instructions | Provider/model-specific prompt variants for Beast/OpenAI, Anthropic, Gemini, and Codex.          | **Mostly present.** `src/prompt/mod.rs` has OpenAI, Anthropic, Gemini, Codex, and Generic prompt branches.                                                                                | Selection is based on model-id string heuristics, not resolved provider kind/model metadata, so OpenAI-compatible or renamed models can get the wrong prompt.                                                              |
 | 2.2  | Environment context block                          | Includes workdir, git status, platform, and date.                                                | **Present.** `SystemPromptComposer::get_environment_context` emits `<env>` with working directory, git-repo flag, platform, and date.                                                     | No material harness gap.                                                                                                                                                                                                   |
-| 2.3  | Tool schemas block                                 | System prompt lists registered tool schemas as JSON.                                             | **Present.** `SystemPromptComposer::with_tool_registry` emits JSON schemas; app and print mode compose prompts with scoped dynamic registries.                                            | Schemas are scoped to visible tools for the current mode, not literally every registered tool; registry ordering is not deterministic because it is backed by a `HashMap`.                                                 |
+| 2.3  | Tool schemas block                                 | System prompt lists registered tool schemas as JSON.                                             | **Changed.** `SystemPromptComposer::with_tool_registry` now emits tool **names** + short guidance only; full schemas go solely via the provider `tools` field (OpenCode/Codex/Grok Build). | Avoids double-paying schema tokens every step; registry ordering is still non-deterministic because it is backed by a `HashMap`.                                                                                            |
 | 2.4  | Custom instructions discovery                      | Walk-up discovery for `AGENTS.md`/`CLAUDE.md` plus global fallback.                              | **Partial.** `src/prompt/rules.rs` walks upward for nearest local `AGENTS.md` or `CLAUDE.md`; global fallback checks `$XDG_CONFIG_HOME/crabcode/AGENTS.md` and `~/.claude/CLAUDE.md`.     | Walk-up does not stop at the git/project root; it does not load OpenCode global instruction locations; it returns the first local match instead of a layered instruction stack.                                            |
 | 2.5  | Available skills XML block                         | Prompt lists discovered skills as `<available_skills>`.                                          | **Present.** `src/prompt/mod.rs` appends skills from `SkillStore`; `src/tools/skill.rs` repeats them in the tool description.                                                             | No material gap beyond discovery/permission gaps listed below.                                                                                                                                                             |
 | 2.6  | Available subagents XML block                      | Prompt lists subagent names/descriptions so the primary agent can use Task.                      | **Mostly present.** `src/prompt/mod.rs` emits visible subagent definitions from `AgentRegistry`, including config-defined agents.                                                         | Missing OpenCode built-in `scout` and `vlm-agent`; hidden agents are intentionally omitted from prompt listings.                                                                                                          |

npm/README.md

@@ -23,7 +23,7 @@ A purely Rust-based AI CLI coding agent with a beautiful terminal UI for interac
   - **Multiple Model Support** - Works w/ the same models.dev support.
   - **Command System** - Intuitive commands: `/sessions`, `/new`, `/connect`, `/models`, `/exit` + custom commands.
   - **Session Management** - Create and manage multiple chat sessions
-  - **Streaming Responses** - Real-time streaming of AI responses (w/ [aisdk.rs](https://aisdk.rs))
+  - **Streaming Responses** - Real-time streaming of AI responses + websockets using OpenAI.
 
 ## Installation
 
@@ -118,9 +118,9 @@ Like any benchmark, please take this with a grain of salt. I have a cherry-picke
 
 | Agent       | Score | Checks | Avg time | Est. tokens | Est. cost |
 | ----------- | ----: | -----: | -------: | ----------: | --------: |
-| 🦀 crabcode |  100% |  19/19 |    21.3s |        2851 |   $0.0103 |
-| 🔲 opencode |  100% |  19/19 |    24.2s |        3837 |   $0.0201 |
-| ⚛️ codex    |  100% |  19/19 |    24.9s |       56172 |   $0.5435 |
+| 🦀 crabcode |  100% |  19/19 |    29.8s |        2768 |   $0.0094 |
+| 🔲 opencode |  100% |  19/19 |    34.9s |        4612 |   $0.0279 |
+| ⚛️ codex    |  100% |  19/19 |    33.7s |       36888 |   $0.3506 |
 
 ## Contributing
 

src/agent/config.rs

@@ -10,6 +10,9 @@ pub struct OpenAIRequestOptions {
     pub default_instructions: Option<String>,
     pub disallow_system_messages: bool,
     pub force_tool_strict_false: bool,
+    /// Sticky prompt-cache routing key (Responses / chat-completions).
+    /// Typically the crabcode session id.
+    pub prompt_cache_key: Option<String>,
 }
 
 #[derive(Debug, Clone, PartialEq, Eq)]
@@ -83,6 +86,8 @@ pub struct LlmSessionConfig {
     pub reasoning_effort: Option<crate::model::reasoning::ReasoningEffort>,
     pub supports_image_input: bool,
     pub openai_options: OpenAIRequestOptions,
+    /// Sticky prompt-cache key for this session (OpenAI/xAI/compatible).
+    pub prompt_cache_key: Option<String>,
 }
 
 #[derive(Debug, Clone, Copy, PartialEq, Eq)]
@@ -131,6 +136,7 @@ mod tests {
             reasoning_effort: None,
             supports_image_input: false,
             openai_options: OpenAIRequestOptions::default(),
+            prompt_cache_key: None,
         }
     }
 

src/agent/subagent.rs

@@ -323,6 +323,13 @@ async fn start_subagent_stream(
             if let Some(effort) = session.reasoning_effort {
                 builder = builder.reasoning_effort(effort.as_str());
             }
+            if let Some(cache_key) = session
+                .prompt_cache_key
+                .as_deref()
+                .or(session.openai_options.prompt_cache_key.as_deref())
+            {
+                builder = builder.prompt_cache_key(cache_key);
+            }
             let provider = builder
                 .build()
                 .map_err(|e| format!("Failed to build OpenAICompatible provider: {}", e))?;
@@ -389,6 +396,13 @@ async fn start_subagent_stream(
             if session.openai_options.force_tool_strict_false {
                 builder = builder.tool_strict_override(false);
             }
+            if let Some(cache_key) = session
+                .prompt_cache_key
+                .as_deref()
+                .or(session.openai_options.prompt_cache_key.as_deref())
+            {
+                builder = builder.prompt_cache_key(cache_key);
+            }
             if !session.openai_options.additional_headers.is_empty() {
                 builder = builder.headers(session.openai_options.additional_headers.clone());
             }
@@ -615,6 +629,7 @@ mod tests {
             reasoning_effort,
             supports_image_input: false,
             openai_options: crate::agent::config::OpenAIRequestOptions::default(),
+            prompt_cache_key: None,
         }
     }
 }

src/aisdk/providers/anthropic.rs

@@ -96,7 +96,7 @@ impl Provider for Anthropic {
     ) -> Result<ProviderStream> {
         let url = format!("{}/v1/messages", self.base_url.trim_end_matches('/'));
 
-        let system_prompts: Vec<serde_json::Value> = messages
+        let mut system_prompts: Vec<serde_json::Value> = messages
             .iter()
             .filter_map(|m| match m {
                 Message::System(s) => Some(serde_json::json!({
@@ -107,6 +107,17 @@ impl Provider for Anthropic {
             })
             .collect();
 
+        // Anthropic prompt caching: mark the last system block ephemeral so the
+        // stable prefix can be reused across tool steps in a session.
+        if let Some(last) = system_prompts.last_mut() {
+            if let Some(obj) = last.as_object_mut() {
+                obj.insert(
+                    "cache_control".to_string(),
+                    serde_json::json!({ "type": "ephemeral" }),
+                );
+            }
+        }
+
         let user_messages: Vec<serde_json::Value> = messages
             .iter()
             .filter_map(|m| match m {

src/aisdk/providers/compatible.rs

@@ -18,6 +18,7 @@ pub struct OpenAICompatible {
     model_name: String,
     provider_name: String,
     reasoning_effort: Option<String>,
+    prompt_cache_key: Option<String>,
 }
 
 impl OpenAICompatible {
@@ -33,6 +34,7 @@ pub struct OpenAICompatibleBuilder {
     model_name: Option<String>,
     provider_name: Option<String>,
     reasoning_effort: Option<String>,
+    prompt_cache_key: Option<String>,
 }
 
 impl OpenAICompatibleBuilder {
@@ -61,6 +63,11 @@ impl OpenAICompatibleBuilder {
         self
     }
 
+    pub fn prompt_cache_key(mut self, key: impl Into<String>) -> Self {
+        self.prompt_cache_key = Some(key.into());
+        self
+    }
+
     pub fn build(self) -> Result<OpenAICompatible> {
         Ok(OpenAICompatible {
             base_url: self
@@ -74,6 +81,7 @@ impl OpenAICompatibleBuilder {
                 .provider_name
                 .unwrap_or_else(|| "openai-compatible".to_string()),
             reasoning_effort: self.reasoning_effort,
+            prompt_cache_key: self.prompt_cache_key,
         })
     }
 }
@@ -134,6 +142,12 @@ impl Provider for OpenAICompatible {
             body["reasoning_effort"] = serde_json::Value::String(effort.clone());
         }
 
+        if let Some(key) = &self.prompt_cache_key {
+            if !key.is_empty() {
+                body["prompt_cache_key"] = serde_json::Value::String(key.clone());
+            }
+        }
+
         let mut request_headers = reqwest::header::HeaderMap::new();
         request_headers.insert(
             reqwest::header::CONTENT_TYPE,

src/aisdk/providers/openai.rs

@@ -28,6 +28,14 @@ const OPENAI_WEBSOCKET_IO_TIMEOUT: Duration = Duration::from_secs(300);
 const OPENAI_WEBSOCKET_STREAM_RETRIES: usize = 1;
 const OPENAI_WEBSOCKET_FAILURES_BEFORE_FALLBACK: usize = 5;
 
+#[async_trait]
+pub trait HttpResponseRetryPolicy: Send + Sync + std::fmt::Debug {
+    async fn retry_headers(
+        &self,
+        status: reqwest::StatusCode,
+    ) -> Option<reqwest::header::HeaderMap>;
+}
+
 #[derive(Debug, Clone)]
 pub struct OpenAI {
     base_url: String,
@@ -42,6 +50,8 @@ pub struct OpenAI {
     default_instructions: Option<String>,
     reasoning_effort: Option<String>,
     responses_websocket: bool,
+    prompt_cache_key: Option<String>,
+    response_retry_policy: Option<Arc<dyn HttpResponseRetryPolicy>>,
     websocket_state: Arc<Mutex<OpenAIWebsocketState>>,
 }
 
@@ -77,6 +87,8 @@ pub struct OpenAIBuilder {
     default_instructions: Option<String>,
     reasoning_effort: Option<String>,
     responses_websocket: bool,
+    prompt_cache_key: Option<String>,
+    response_retry_policy: Option<Arc<dyn HttpResponseRetryPolicy>>,
 }
 
 impl OpenAIBuilder {
@@ -140,6 +152,16 @@ impl OpenAIBuilder {
         self
     }
 
+    pub fn prompt_cache_key(mut self, key: impl Into<String>) -> Self {
+        self.prompt_cache_key = Some(key.into());
+        self
+    }
+
+    pub fn response_retry_policy(mut self, policy: Arc<dyn HttpResponseRetryPolicy>) -> Self {
+        self.response_retry_policy = Some(policy);
+        self
+    }
+
     pub fn build(self) -> Result<OpenAI> {
         let base_url = self
             .base_url
@@ -174,6 +196,8 @@ impl OpenAIBuilder {
             default_instructions: self.default_instructions,
             reasoning_effort: self.reasoning_effort,
             responses_websocket: self.responses_websocket,
+            prompt_cache_key: self.prompt_cache_key,
+            response_retry_policy: self.response_retry_policy,
             websocket_state: Arc::new(Mutex::new(OpenAIWebsocketState::default())),
         })
     }
@@ -364,9 +388,9 @@ impl Provider for OpenAI {
             ))
             .build()
             .map_err(|e| Error::Provider(format!("Failed to build client: {}", e)))?;
-        let response = client
+        let mut response = client
             .post(&url)
-            .headers(request_headers)
+            .headers(request_headers.clone())
             .json(&body)
             .send()
             .await
@@ -379,6 +403,28 @@ impl Provider for OpenAI {
                 )))
             })?;
 
+        if let Some(policy) = &self.response_retry_policy {
+            if let Some(retry_headers) = policy.retry_headers(response.status()).await {
+                request_headers.extend(retry_headers);
+                response = client
+                    .post(&url)
+                    .headers(request_headers)
+                    .json(&body)
+                    .send()
+                    .await
+                    .map_err(|err| {
+                        Error::RetryableProvider(RetryError::from_message(
+                            format_openai_request_error(
+                                "send_after_response_retry_policy",
+                                &url,
+                                &err,
+                                Some(&request_diagnostics),
+                            ),
+                        ))
+                    })?;
+            }
+        }
+
         if !response.status().is_success() {
             let status = response.status();
             let response_url = sanitized_url(response.url());
@@ -530,6 +576,12 @@ impl OpenAI {
             body["reasoning"] = serde_json::json!({ "effort": effort });
         }
 
+        if let Some(key) = &self.prompt_cache_key {
+            if !key.is_empty() {
+                body["prompt_cache_key"] = serde_json::Value::String(key.clone());
+            }
+        }
+
         body
     }
 
@@ -2100,6 +2152,27 @@ mod tests {
         assert!(body.get("tools").is_none());
         assert!(body.get("tool_choice").is_none());
         assert!(body.get("parallel_tool_calls").is_none());
+        assert!(body.get("prompt_cache_key").is_none());
+    }
+
+    #[test]
+    fn responses_body_includes_prompt_cache_key_and_store_false() {
+        let provider = OpenAI::builder()
+            .base_url("https://api.x.ai")
+            .api_key("test-key")
+            .model_name("grok-composer-2.5-fast")
+            .prompt_cache_key("session-abc")
+            .store_override(false)
+            .build()
+            .unwrap();
+
+        let body = provider.build_responses_body(
+            vec![serde_json::json!({"role": "user", "content": "hi"})],
+            &[],
+        );
+
+        assert_eq!(body["prompt_cache_key"], "session-abc");
+        assert_eq!(body["store"], false);
     }
 
     #[test]