Experimental Native Protocol Rewrite

.env.example

@@ -333,6 +333,81 @@
 # Default: false
 # STREAM_RETRY_ON_REASONING_ONLY=false
 
+# --- Optional Structured Config File ---
+# Optional JSON config for structured experimental settings such as fallback
+# groups, model routes, advisory pricing, streaming observability, and field
+# cache rules. Environment variables override JSON config. Do not put API keys,
+# OAuth tokens, bearer tokens, or authorization headers in this file.
+# Supported top-level sections include: routing, pricing, streaming, retry,
+# responses, field_cache, and providers. The providers section is for non-secret
+# provider tuning only; credentials must stay in env or provider credential files.
+# LLM_PROXY_CONFIG_FILE=./config/llm-proxy.json
+
+# --- Fallback Groups and Model Routes ---
+# Ordered fallback groups try targets in order when the previous target fails
+# with a retryable provider/category error. Execution suffixes:
+#   @auto              Let the provider choose custom/native/LiteLLM behavior
+#   @native            Require native protocol execution
+#   @custom            Require provider custom execution
+#   @litellm_fallback  Explicitly use LiteLLM fallback
+# FALLBACK_GROUPS=code_chain
+# FALLBACK_GROUP_CODE_CHAIN=codex/gpt-5.1-codex@native,openai/gpt-5.1@litellm_fallback
+# MODEL_ROUTE_CODE=group:code_chain
+
+# --- Provider Cooldown Activation ---
+# Provider-level cooldown is conservative and only intended for large/global
+# retry-after events, not every per-credential quota error. Model-capacity
+# errors can start a model-scoped cooldown without blocking unrelated models.
+# PROVIDER_COOLDOWN_MIN_SECONDS=10
+# PROVIDER_COOLDOWN_DEFAULT_SECONDS=30
+# PROVIDER_COOLDOWN_ON_QUOTA=false
+# Repeated transient provider/model failures can increase bounded backoff.
+# PROVIDER_BACKOFF_WINDOW_SECONDS=60
+# PROVIDER_BACKOFF_THRESHOLD=3
+# PROVIDER_BACKOFF_BASE_SECONDS=0  # 0/unset means use provider cooldown default
+# PROVIDER_BACKOFF_MAX_SECONDS=300
+# FAILURE_HISTORY_MAX_ENTRIES=200
+
+# --- Responses API Store Policy ---
+# Responses are stored in-memory by default. Use provider_cache for durable JSON
+# storage via the existing provider-cache layer. TTL/max limits are disabled
+# when unset or <= 0. Failed stream responses are stored by default; in-progress
+# streaming state is disabled unless explicitly enabled.
+# RESPONSES_STORE_BACKEND=memory
+# RESPONSES_STORE_CACHE_NAME=responses
+# RESPONSES_STORE_CACHE_PREFIX=responses
+# RESPONSES_STORE_CACHE_DIR=
+# RESPONSES_STORE_CACHE_MEMORY_TTL_SECONDS=3600
+# RESPONSES_STORE_CACHE_DISK_TTL_SECONDS=172800
+# RESPONSES_STORE_TTL_SECONDS=0
+# RESPONSES_STORE_MAX_ITEMS=0
+# RESPONSES_STORE_FAILED=true
+# RESPONSES_STORE_IN_PROGRESS=false
+
+# --- Streaming Observability ---
+# Stream lifecycle metrics are traced by default. TTFB/stall timeout values are
+# active when set to >0; keep at 0 to disable. Heartbeats are SSE comments and
+# do not count as visible output.
+# STREAM_TRACE_METRICS=true
+# STREAM_TTFB_TIMEOUT_SECONDS=0
+# STREAM_STALL_TIMEOUT_SECONDS=0
+# STREAM_HEARTBEAT_INTERVAL_SECONDS=0
+# STREAM_HEARTBEAT_SECONDS=0  # Legacy alias
+# STREAM_CANCEL_UPSTREAM_ON_DISCONNECT=true
+
+# --- Advisory Model Pricing ---
+# Per-token advisory prices used only when providers do not report actual cost.
+# Precedence: skip-cost provider setting > provider-reported cost/SSE cost event
+# > provider explicit pricing > env pricing > JSON pricing > LiteLLM metadata.
+# Streaming providers can report actual cost with `: cost {...}` comments or
+# `event: cost` frames.
+# Env names sanitize provider/model by replacing non-alphanumerics with `_`.
+# MODEL_PRICE_OPENAI_GPT_5_1_INPUT=0.000001
+# MODEL_PRICE_OPENAI_GPT_5_1_OUTPUT=0.00001
+# MODEL_PRICE_OPENAI_GPT_5_1_CACHE_READ=0.0000001
+# MODEL_PRICE_OPENAI_GPT_5_1_CACHE_WRITE=0.000001
+# MODEL_PRICE_OPENAI_GPT_5_1_REASONING=0.00001
+
 # ------------------------------------------------------------------------------
 # | [ADVANCED] HTTP Timeout Configuration                                       |
 # ------------------------------------------------------------------------------

docs/experimental/00-master-plan.md

@@ -0,0 +1,132 @@
+# Experimental Native Protocol Roadmap
+
+This branch is for a long-running experimental rewrite that makes native protocol support the first-class extension point of `rotator_library`, while preserving the existing credential rotation, quota, fair-cycle, session tracking, and provider plugin strengths.
+
+## Operating Rules
+
+- Work only on the `experimental` branch.
+- Keep all repository work inside `C:\Projects\test\LLM-API-Key-Proxy` and child paths.
+- Treat commits as checkpoints. A phase may contain many commits.
+- Commit messages must include a body describing what changed, why, tests run, and follow-up considerations.
+- Do not commit phase reports written for the user unless explicitly requested. Planning docs under `docs/experimental/` are committed.
+- Before each phase implementation, first produce a fresh exhaustive phase plan in conversation text, based on the current code state. Only after that plan is settled should it be written to `docs/experimental/phase-N-*.md`.
+- After each phase implementation, call both `explore` and `explore-heavy` agents to review the work against the phase plan, external reference areas, and current proxy behavior. Fix findings and re-review as needed.
+- Keep LiteLLM as a fallback path for protocols/providers that are not natively covered yet. Native protocol support should be preferred when available.
+
+## Strategic Goal
+
+The target architecture is:
+
+```text
+client API request
+  -> protocol parse into unified representation
+  -> field-cache injection
+  -> adapter chain
+  -> provider override hooks
+  -> provider-native request build
+  -> provider execution and credential rotation
+  -> provider-native response/stream parse
+  -> field-cache extraction
+  -> adapter chain
+  -> protocol formatting for the client
+  -> transaction logging for every transform state
+```
+
+Providers should be able to declare an existing protocol and only override the parts that are genuinely provider-specific. A custom provider should usually be configurable through protocol choice, adapters, field-cache rules, auth strategy, and model options rather than requiring a large bespoke provider implementation.
+
+## Priority Order
+
+1. Native protocol foundations, unified types, transformers, adapters, and field-cache rules.
+2. OpenAI Responses API support, including future WebSocket extension points.
+3. Provider work following the protocol layer: Claude Code, Codex, Copilot, Antigravity, and Gemini CLI parity review.
+4. Routing and fallback groups, with optional target-group selectors later.
+5. Retry, provider/model cooldown, and failover cleanup.
+6. Protocol-aware quota, usage, and cost normalization.
+7. Streaming library hardening: SSE now, WebSocket-ready later.
+8. Config polish using `.env` and optional JSON. No SQLite dependency for now.
+9. Extensive staged tests and review-agent verification.
+
+## Non-Goals For This Branch
+
+- Do not make the proxy a full multi-user admin product yet.
+- Do not require SQLite or Postgres for the main feature set.
+- Do not remove LiteLLM before native coverage exists.
+- Do not replace the existing `UsageManager`, fair-cycle, custom caps, or evidence-based `SessionTracker`.
+- Do not port frontend/UI work from the external reference gateway.
+
+## Current Strengths To Preserve
+
+- Credential-level rotation and priority-aware selection.
+- Fair cycle and custom caps.
+- Windowed quota tracking and quota groups.
+- Evidence-based session tracking with compaction handling.
+- Provider plugin discovery.
+- Gemini CLI provider behavior unless a reviewed change is clearly better.
+- Resilient file/JSON state writing.
+- Dynamic OpenAI-compatible provider discovery.
+
+## Reference Gateway Ideas To Import Carefully
+
+- Unified protocol/transformer style.
+- Adapter registry and configurable provider/model adapters.
+- Target groups and direct routing syntax, adapted into fallback-first routing.
+- Responses API transformer and storage concepts.
+- Stream TTFB/stall detection concepts, implemented with Python-native async primitives.
+- Provider/model cooldown and retry-history concepts.
+- Usage/cost normalization and provider-reported cost extraction.
+- Broader provider support patterns for Claude Code, Codex, Copilot, and Antigravity.
+
+## Phase Index
+
+1. Protocol Core.
+2. Transform Pass Logging.
+3. Adapter and Field Cache System.
+4. Responses API and WebSocket-Ready Transport Shape.
+5. Provider Protocol Overhaul.
+6. Routing and Fallback Groups.
+7. Retry/Cooldown/Failover Cleanup.
+8. Streaming Library Upgrade.
+9. Usage, Quota, and Cost Accuracy.
+10. Config Polish.
+
+Each phase may be subdivided if implementation scope becomes too large.
+
+## Completeness Matrix
+
+This matrix exists so the branch does not lose any requested scope while phases evolve. The phase plans are still refreshed before implementation, but every item below must remain accounted for.
+
+| Requested area | Planned coverage |
+| --- | --- |
+| Protocols are priority #1 | Phases 1 and 4 create native protocol foundations and Responses support before provider work. |
+| Protocols are bases, not gospel | Phase 1 requires override-friendly protocol methods, subclassing, copy/mutate registration, and provider-specific overrides. |
+| Move away from LiteLLM | Phase 1 adds a `litellm_fallback` protocol path; later providers should prefer native protocols and use LiteLLM only for unsupported coverage. |
+| Add protocols automatically like providers | Phase 1 adds protocol auto-discovery and registry behavior modeled after provider discovery. |
+| Cover current providers and reference providers | Phase 1 protocols must cover shapes used by current providers; Phase 5 covers Claude Code, Codex, Copilot, Antigravity, and Gemini CLI parity. |
+| Responses API is very needed | Phase 4 is dedicated to Responses, `previous_response_id`, storage, SSE, and WebSocket-ready transport shape. |
+| WebSocket support later | Phases 1, 4, and 8 require transport separation so WebSocket can be added without rewriting protocol logic. |
+| Adapters/transformers tied to protocols | Phases 1, 2, and 3 define protocol parse/build plus transform tracing, adapter registry, and field-cache rules. |
+| Cache and return provider fields | Phase 3 implements configurable extraction/injection rules for request, response, and stream fields with scope and mode controls. |
+| Reasoning content and similar fields | Phase 3 explicitly covers reasoning content, thinking signatures, prompt cache keys, response IDs, and provider session IDs. |
+| Return all possible or last user/assistant use | Phase 3 modes include `last`, `all`, `last_user_turn`, `last_assistant_turn`, and `per_tool_call`. |
+| Per-model custom provider behavior | Phases 3, 5, and 10 cover provider/model field cache rules, adapters, model options, and optional JSON config. |
+| Transaction logging after every transform | Phase 2 adds ordered request, response, and stream transform trace passes and integrates them with transaction logging. |
+| Comments, docstrings, and key decisions | All implementation phases require docstrings for public abstractions and comments for non-obvious transform, protocol, and future-extension decisions. |
+| Providers are priority #2 | Phase 5 follows protocol foundations with Claude Code, Codex, Copilot, Antigravity, and Gemini CLI parity review. |
+| Antigravity comparison | Phase 5 explicitly compares the reference Antigravity behavior against `src/rotator_library/providers/_retired/`. |
+| Routing is interesting | Phase 6 implements fallback chains first, with target-group selectors later if useful. |
+| Fallback groups preferred over target groups | Phase 6 starts with ordered fallback groups and only adds target-group-style selectors after that base works. |
+| Retry/cooldown/failover cleanup | Phase 7 makes provider/model cooldown real, adds retry history, backoff, retry-after precedence, and success reset. |
+| Quota/usage/cost improvements | Phase 9 adds protocol-aware normalizers, provider-reported cost extraction, structured cost fields, and checker abstractions while keeping existing usage engines. |
+| Streaming as library capability | Phase 8 hardens streaming below the proxy route layer with TTFB, TTFT, stall detection, cancellation, and transport-aware stream events. |
+| Config via env/json, no SQLite | Phase 10 adds optional JSON config with env overrides and validation. SQLite remains out of scope. |
+| Multi-user proxy later | The branch keeps multi-user/admin features as a future expansion and only preserves extension points where natural. |
+| Exhaustive tests in stages | Every phase requires tests alongside implementation and phase-end review by both `explore` and `explore-heavy`. |
+| Reports are for the user, not git | `06-phase-workflow.md` says planning docs are committed, but phase reports are not committed by default. |
+
+## Code Quality Expectations
+
+- Public protocol, adapter, transport, field-cache, and provider-extension classes must have docstrings that explain intent, override points, and future expansion hooks.
+- Non-obvious transformations must have comments explaining why data is changed, preserved, reordered, or intentionally dropped.
+- Lossy protocol conversions must be documented at the conversion site.
+- Future WebSocket, target-group, and multi-user extension seams should be noted in comments where they affect today's design.
+- Tests should prefer golden fixtures for protocol shapes and focused unit tests for transform edge cases.

docs/experimental/01-protocol-architecture.md

@@ -0,0 +1,132 @@
+# Native Protocol Architecture
+
+Protocols are reusable bases, not rigid gospel. Providers can subclass, wrap, copy, or override protocol behavior when a provider deviates from an otherwise standard protocol.
+
+## Why Protocols First
+
+The current code relies heavily on LiteLLM and provider-specific transforms. That works, but it makes new protocols hard to reason about and makes debugging transformations difficult. The experimental goal is to make a provider mostly declarative:
+
+```text
+provider = protocol + auth + adapters + field cache rules + model options + quota behavior
+```
+
+If a provider needs custom behavior, it should override a narrow protocol method instead of forcing an entirely bespoke request path.
+
+## Auto-Discovery
+
+Protocols should follow the provider plugin style:
+
+- protocol modules live under `src/rotator_library/protocols/`.
+- modules register concrete protocol classes by name.
+- a registry exposes names such as `openai_chat`, `anthropic_messages`, `gemini`, `responses`, and `litellm_fallback`.
+- third-party or local protocol modules can be added later with minimal registry changes.
+
+## Core Types
+
+The unified representation should be explicit enough to cover all existing providers and the external reference protocols without losing important data.
+
+Suggested types:
+
+- `UnifiedRequest`
+- `UnifiedResponse`
+- `UnifiedStreamEvent`
+- `UnifiedMessage`
+- `ContentBlock`
+- `ToolDefinition`
+- `ToolCall`
+- `ToolResult`
+- `ReasoningBlock`
+- `Usage`
+- `CostDetails`
+- `ProtocolMetadata`
+
+These types should retain unknown provider-specific metadata in explicit extension dictionaries instead of dropping it. Robustness matters more than a narrow perfect schema.
+
+## Protocol Interface
+
+The base protocol should provide default methods that can be overridden:
+
+- `parse_request(raw_request, context) -> UnifiedRequest`
+- `build_request(unified_request, context) -> raw_provider_request`
+- `parse_response(raw_response, context) -> UnifiedResponse`
+- `format_response(unified_response, context) -> raw_client_response`
+- `parse_stream_event(raw_event, context) -> UnifiedStreamEvent`
+- `format_stream_event(unified_event, context) -> raw_stream_payload`
+- `extract_usage(raw_or_unified, context) -> Usage | None`
+- `supports_transport(transport_name) -> bool`
+
+Provider-specific overrides should receive context that includes provider name, model, credential identity, source protocol, target protocol, request ID, and session tracking information.
+
+## Initial Protocols
+
+### OpenAI Chat
+
+Must support:
+
+- chat completions request/response.
+- stream chunks.
+- tools and tool calls.
+- function-call legacy shapes.
+- reasoning fields from OpenAI-compatible providers.
+- cached token and reasoning token usage details.
+
+### Anthropic Messages
+
+Must support:
+
+- messages request/response.
+- system content extraction.
+- text, image, thinking, redacted thinking, tool_use, tool_result blocks.
+- stream lifecycle events.
+- count_tokens path later if needed.
+
+### Gemini
+
+Must support:
+
+- generateContent and streamGenerateContent shapes.
+- content parts.
+- functionCall/functionResponse.
+- thought signatures.
+- safety settings passthrough without unsafe auto-injection.
+- Google/Gemini usage metadata.
+
+### Responses
+
+Must support:
+
+- OpenAI Responses request/response.
+- `previous_response_id`.
+- output items.
+- event streams.
+- storage-friendly response objects.
+- future WebSocket transport.
+
+### LiteLLM Fallback
+
+Must preserve existing behavior for providers/protocols not yet native. This path should be explicit and transaction-logged as a fallback, not hidden.
+
+## Transport Separation
+
+Protocol formatting must not be tied only to HTTP SSE. Define a transport boundary so the same unified stream events can be emitted through:
+
+- non-streaming HTTP JSON.
+- HTTP SSE.
+- future WebSocket.
+
+The Responses phase should leave clear extension points for WebSocket even if WebSocket is implemented later.
+
+## Error Handling
+
+Protocols should preserve provider error bodies where safe, but format client-facing errors consistently. Parsing errors should include transform-pass names and request IDs to make transaction logs useful.
+
+## Docstrings And Comments
+
+Protocol code should include docstrings explaining:
+
+- which external API shape it models.
+- what fields are intentionally preserved in metadata.
+- where provider overrides are expected.
+- future expansion hooks.
+
+Comments should explain non-obvious transformations, especially lossy conversions between protocols.