Conversation
Introduce AI-native runtime components: new workspace crates rustapi-agent, rustapi-ai, rustapi-llm, rustapi-context, rustapi-memory, and rustapi-tools. Implements a step-based deterministic agent engine (engine, context, step, planner, replay, error) with comprehensive unit and integration tests, closure-step helpers, replay/divergence detection, and planner primitives (StaticPlanner, ReActPlanner). Update workspace Cargo.toml to include the new crates and add documentation pages and examples for the AI runtime.
There was a problem hiding this comment.
Pull request overview
This pull request introduces a comprehensive AI-native runtime to RustAPI, adding six new internal crates that provide agent orchestration, LLM routing, tool execution, and memory management capabilities. The runtime is designed as a modular, pluggable system with the rustapi-ai crate serving as a unified facade.
Changes:
- Adds AI runtime infrastructure with 6 new crates: context, memory, tools, agent, llm, and ai (facade)
- Implements step-based agent execution with planning, branching, and replay capabilities
- Provides cost tracking, execution tracing, and event-based observability
- Integrates with HTTP middleware via extractors (AiCtx, AiRt)
Reviewed changes
Copilot reviewed 57 out of 58 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| Cargo.toml | Adds 6 new AI crates to workspace with chrono dependency |
| crates/rustapi-context/* | Request context with trace trees, cost tracking, event bus |
| crates/rustapi-memory/* | Pluggable memory abstraction with InMemoryStore |
| crates/rustapi-tools/* | Tool registry and DAG execution graph |
| crates/rustapi-llm/* | LLM router with cost-aware fallback and structured output |
| crates/rustapi-agent/* | Step-based agent engine with planning and replay |
| crates/rustapi-ai/* | Unified facade and HTTP middleware integration |
| crates/rustapi-rs/* | Feature flags and prelude exports for AI runtime |
| docs/cookbook/* | Documentation for the AI runtime architecture |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| impl Drop for SpanGuard { | ||
| fn drop(&mut self) { | ||
| // If not explicitly completed/failed, mark as error and record. | ||
| if self.node.status == TraceStatus::InProgress { | ||
| self.node.fail("span dropped without completion"); | ||
| } | ||
| self.tree.add_root_child(self.node.clone()); | ||
| } | ||
| } |
There was a problem hiding this comment.
The SpanGuard's Drop implementation unconditionally adds the node to the tree even after explicit complete() or fail() calls, which use std::mem::forget to prevent double-insertion. However, if a panic occurs during step execution, the span will be added with an error status via Drop. This is correct behavior for panic safety, but the comment on line 291-293 should clarify that this handles the panic case specifically, as the "dropped without completion" message might be misleading during panics.
| pub fn record(&self, delta: &CostDelta) -> Result<(), ContextError> { | ||
| self.input_tokens | ||
| .fetch_add(delta.input_tokens, Ordering::Relaxed); | ||
| self.output_tokens | ||
| .fetch_add(delta.output_tokens, Ordering::Relaxed); | ||
| self.total_cost_micros | ||
| .fetch_add(delta.cost_micros, Ordering::Relaxed); | ||
| self.api_calls.fetch_add(1, Ordering::Relaxed); | ||
|
|
||
| self.check_budget() | ||
| } |
There was a problem hiding this comment.
The CostTracker::record method applies the delta first, then checks the budget (line 144). This "fail-open" behavior means the first request that exceeds the budget will have its cost recorded before failing. While this is mentioned in the comment on line 134, it could lead to slight budget overruns. Consider checking the budget before applying the delta, or document this trade-off more prominently in the public API docs.
| ToolNode::Parallel { id: _, nodes } => { | ||
| // For parallel execution, we need to collect results. | ||
| // Since outputs is &mut, we execute sequentially here for safety. | ||
| // A production implementation would use JoinSet with per-node output maps. | ||
| let mut handles = Vec::new(); | ||
| let registry = registry.clone(); | ||
| let ctx = ctx.clone(); | ||
|
|
||
| for child_node in nodes { | ||
| let reg = registry.clone(); | ||
| let c = ctx.clone(); | ||
| let node = child_node.clone(); | ||
| handles.push(tokio::spawn(async move { | ||
| let mut local_outputs = HashMap::new(); | ||
| // We create a temporary graph to execute the child node. | ||
| let graph = ToolGraph::new("parallel_child", node.clone()); | ||
| match graph | ||
| .execute_node(&node, ®, &c, &mut local_outputs) | ||
| .await | ||
| { | ||
| Ok(()) => Ok(local_outputs), | ||
| Err(e) => Err(e), | ||
| } | ||
| })); | ||
| } | ||
|
|
||
| for handle in handles { | ||
| match handle.await { | ||
| Ok(Ok(local_outputs)) => { | ||
| outputs.extend(local_outputs); | ||
| } | ||
| Ok(Err(e)) => return Err(e), | ||
| Err(e) => { | ||
| return Err(ToolError::internal(format!("Join error: {e}"))); | ||
| } | ||
| } | ||
| } | ||
| Ok(()) | ||
| } |
There was a problem hiding this comment.
The parallel execution implementation has a potential race condition. Each spawned task creates a temporary graph and local outputs HashMap, but the comment mentions this is a simplified implementation. In production, the shared mutable outputs HashMap could cause issues if accessed concurrently without proper synchronization. The current approach serializes parallel execution by extending outputs sequentially after join, which defeats the purpose of parallelism for the output collection phase.
| pub struct ClosureTool<F> | ||
| where | ||
| F: Fn(&RequestContext, serde_json::Value) -> std::pin::Pin<Box<dyn std::future::Future<Output = Result<ToolOutput, ToolError>> + Send + '_>> | ||
| + Send | ||
| + Sync | ||
| + 'static, | ||
| { | ||
| name: String, | ||
| description: String, | ||
| parameters: serde_json::Value, | ||
| handler: F, | ||
| } |
There was a problem hiding this comment.
The ClosureTool and ClosureStep function signatures require manually constructing Pin<Box> which is extremely verbose and error-prone. Consider using async_trait on the closure trait bounds or providing a macro to simplify this. The current API forces users to write Box::pin(async move { ... }) every time, which is not ergonomic.
|
@copilot open a new pull request to apply changes based on the comments in this thread |
- trace.rs: Clarify SpanGuard::drop comment to explain it handles panics - cost.rs: Pre-check budget before applying delta to avoid fail-open overrun - graph.rs: Use JoinSet for parallel execution; collect results as tasks finish - tool.rs: Add ToolFuture<'a> type alias to simplify ClosureTool signatures" Co-authored-by: Tuntii <121901995+Tuntii@users.noreply.github.com>
Apply review feedback: trace panic safety, cost pre-check, parallel JoinSet, ToolFuture alias
SUM
This pull request introduces the new AI-native runtime to the RustAPI project, adding a suite of internal crates for agent orchestration, LLM routing, tool execution, and memory management. It also updates the documentation to reflect these new capabilities and reorganizes the project structure and dependencies accordingly.
Key changes:
AI-Native Runtime Integration
rustapi-context,rustapi-memory,rustapi-tools,rustapi-agent,rustapi-llm, andrustapi-aiinCargo.toml. These crates form the foundation for building LLM-powered backends in RustAPI.rustapi-agentcrate with its ownCargo.toml, specifying dependencies on other AI runtime crates and common libraries.Agent Context Implementation
AgentContextinrustapi-agent/src/context.rs, providing a per-execution context for agent steps. This struct manages access to the request context, tool registry, memory store, agent-local state, step tracking, and streaming yields, enabling step-based and deterministic agent execution.Documentation Overhaul
README.mdto document the new AI-native runtime, its architecture, usage examples, feature flags, and comparison with other frameworks. The documentation now details the responsibilities of each new crate and how to enable and use the AI runtime in RustAPI projects. [1] [2]These changes lay the groundwork for RustAPI's integrated support for LLMs and agent-based workflows, making Rust a first-class choice for AI backend development.
Type of Change
Checklist
api/public/*snapshots changed, this PR hasbreakingorfeaturelabel