refactor!: extract shared agent system engine

[shadaj]

Stack

This is PR 1 of 2. Review and merge this PR first. The additive high-level API is split into #92, which is based on this branch.

Summary

Extract the daemon's agent execution machinery into a shared infinity_agent_core::system engine, migrate the daemon and Lambda runtimes onto it, and extract the protocol components those embeddings share.

This PR intentionally stops at the engine and embedding boundary. Static builder conveniences, local MCP/RAP tool-set adapters, ThreadHandle, launcher mode, per-thread launch configuration, and the new usage guides are introduced by #92.

Review guide

Most of this diff is relocation. The old daemon thread_worker.rs and agent_loop.rs loops, plus the old core batch processor, are replaced by focused core modules. The migrated daemon tests in system/tests.rs and their unchanged snapshots are the fidelity evidence for the port.

Shared engine

• system/local/driver.rs ports the worker loop: batching, cancellation, deferral, compaction, tool-result waiting, and idle exit.
• system/thread.rs owns the completion pipeline, commit ordering, observer calls, config resolution, and tool dispatch.
• system/local/router.rs owns input routing, driver spawning, event admission, subscriptions, lifecycle notifications, and shutdown.
• system/builder.rs exposes embedding-oriented configuration, local observer startup, and batch-shaped AgentSystem::step for serverless platforms.
• stores.rs centralizes ancestor, compaction, deduplication, snapshot, and restore behavior. The daemon store wraps it with persisted extras.

Behavior worth reviewing closely

• Split event admission: event-style input cannot create or resume a thread. Before spawning a driver, the router asks ConversationStore::thread_exists whether the exact root or child target exists, then asks StateStore::is_thread_stopped whether the embedding has explicitly stopped it. Unknown and stopped targets are dropped. User text bypasses both checks because it creates new threads and resumes stopped ones. Store failures fail open so a transient read cannot discard a real tool result.
• Lifecycle-owned session state: daemon input submission now always enqueues and contains no session-status policy. A Live lifecycle event resolves the thread's root session and clears both its idle and stopped flags. This applies equally to root and child activity and makes lifecycle the single source of runtime activity state. Event-style input for a stopped session never produces Live because router admission drops it first.
• Lifecycle and subscription-aware activity: RunningSystem::thread_lifecycle emits Live and Idle at the same points the router mutates its live set. The daemon keeps subscribed threads active for RAP server lifetime even though their drivers may idle.
• Idle-exit races: drivers and the router share a LocalSet; the final no-await channel checks make exit atomic with respect to routing. Input either reaches the existing driver or observes closed channels and causes a respawn.
• Observer ordering: history sync completes before CompletionFinished and tool dispatch. Failed dispatch enqueues an error tool result so the agent can recover.
• Batch-shaped step mode: one delivery is partitioned by group_id, deferral is applied sequentially per thread, and thread slices run concurrently.

Protocol foundations

• infinity-rap-bridge provides callback conversion, prepared deduplication IDs, callback serving, and view-update forwarding for daemon and Lambda embeddings.
• infinity-mcp-bridge provides lazy stdio and Streamable HTTP clients plus canonical metadata and dispatch used by the daemon proxy.
• invoke_rap_tool is the shared RAP invocation path used by core, Lambda, and daemon tools, including a uniform non-2xx error result.
• Application-facing RapToolSet and McpToolSet adapters are deferred to feat(infinity-agent-core)!: add high-level agent system API #92.

Production migrations

• Daemon: one daemon-lifetime agent system replaces per-session loops and restart-on-send behavior. RAP servers boot lazily from thread configuration and follow lifecycle/subscription activity. All input paths enqueue through the same router; lifecycle transitions own stopped/idle state updates.
• Lambda: a batch is processed through one AgentSystem::step; per-session tools resolve through ThreadConfigSource; the Rust rap-receiver uses shared callback conversion and replaces the JavaScript receiver. DSQL thread-hierarchy records provide exact event-target existence checks.
• CLI: DisplayEvent moves to the CLI because rendering is no longer a core engine concern.

Diff orientation

The new core system modules include the migrated daemon fidelity tests and shared fixtures. The daemon itself shrinks substantially because agent_loop.rs and thread_worker.rs are removed. Application conveniences and the documentation rewrite are kept out of this PR so reviewers can focus on the engine and production migrations.

Validation

• cargo fmt --all -- --check
• cargo clippy --workspace --all-targets -- -D warnings
• cargo test --workspace

Breaking changes

• Removes batch_processor (process_batch, process_input_item, and core DisplayEvent) in favor of AgentSystem::step and observer-based local execution.
• Thread is internal and AgentSystem is not Clone.
• ThreadObserver replaces inline daemon persistence/display hooks; it has no on_commit.
• EventCollector::take returns (thread_id, event) pairs.
• ConversationStore gains thread_exists, which checks exact root or child records without creating them.
• StateStore gains the provided is_thread_stopped policy hook. User text may resume stopped threads; event-style input may not.
• Builder tools are stored as Rc; Tool gains defaulted is_passive.
• ToolContext and its builder lose input_queue_arn.
• Resident runtime types such as RunningSystem, SubscribeHandle, ChannelSender, and ChannelSendError live under system::local.
• The daemon no longer exports its old worker/loop/session implementation modules, sleep/RAP tool wrappers, or boot_rap_servers.
• SessionManager::send_input no longer accepts a thread ID or user_driven flag and performs no status admission; router admission and lifecycle events own those concerns.
• rap_callback::start_callback_server is replaced by infinity_daemon::launch_session_manager; callback serving accepts a RapCallbackBridge.
• SessionManager::switch_model accepts the requester's sender and returns Result<(), String>; SharedSessionManager is Rc.
• CLI DisplayEvent lives in infinity_agent_cli::display.

[cloudflare-workers-and-pages]

Deploying infinity with    Cloudflare Pages

Latest commit:	2d27833
Status:	✅  Deploy successful!
Preview URL:	https://d68c57fd.infinity-dc7.pages.dev
Branch Preview URL:	https://sandbox-0f86f15a-4591-435a-b.infinity-dc7.pages.dev

View logs

[MingweiSamuel]

Should this be a https://docs.rs/tokio-util/latest/tokio_util/sync/struct.CancellationToken.html

[MingweiSamuel]

I suppose Infinity could wrap in RefCell itself, if this is always expected to be the case