Experimental Native Protocol Rewrite

[Mirrowel]

Experimental Native Protocol Roadmap

This branch is for a long-running experimental rewrite that makes native protocol support the first-class extension point of rotator_library, while preserving the existing credential rotation, quota, fair-cycle, session tracking, and provider plugin strengths.

Operating Rules

• Work only on the experimental branch.
• Keep all repository work inside C:\Projects\test\LLM-API-Key-Proxy and child paths.
• Treat commits as checkpoints. A phase may contain many commits.
• Commit messages must include a body describing what changed, why, tests run, and follow-up considerations.
• Do not commit phase reports written for the user unless explicitly requested. Planning docs under docs/experimental/ are committed.
• Before each phase implementation, first produce a fresh exhaustive phase plan in conversation text, based on the current code state. Only after that plan is settled should it be written to docs/experimental/phase-N-*.md.
• After each phase implementation, call both explore and explore-heavy agents to review the work against the phase plan, external reference areas, and current proxy behavior. Fix findings and re-review as needed.
• Keep LiteLLM as a fallback path for protocols/providers that are not natively covered yet. Native protocol support should be preferred when available.

Strategic Goal

The target architecture is:

client API request
  -> protocol parse into unified representation
  -> field-cache injection
  -> adapter chain
  -> provider override hooks
  -> provider-native request build
  -> provider execution and credential rotation
  -> provider-native response/stream parse
  -> field-cache extraction
  -> adapter chain
  -> protocol formatting for the client
  -> transaction logging for every transform state

Providers should be able to declare an existing protocol and only override the parts that are genuinely provider-specific. A custom provider should usually be configurable through protocol choice, adapters, field-cache rules, auth strategy, and model options rather than requiring a large bespoke provider implementation.

Priority Order

1. Native protocol foundations, unified types, transformers, adapters, and field-cache rules.
2. OpenAI Responses API support, including future WebSocket extension points.
3. Provider work following the protocol layer: Claude Code, Codex, Copilot, Antigravity, and Gemini CLI parity review.
4. Routing and fallback groups, with optional target-group selectors later.
5. Retry, provider/model cooldown, and failover cleanup.
6. Protocol-aware quota, usage, and cost normalization.
7. Streaming library hardening: SSE now, WebSocket-ready later.
8. Config polish using .env and optional JSON. No SQLite dependency for now.
9. Extensive staged tests and review-agent verification.

Non-Goals For This Branch

• Do not make the proxy a full multi-user admin product yet.
• Do not require SQLite or Postgres for the main feature set.
• Do not remove LiteLLM before native coverage exists.
• Do not replace the existing UsageManager, fair-cycle, custom caps, or evidence-based SessionTracker.
• Do not port frontend/UI work from the external reference gateway.

Current Strengths To Preserve

• Credential-level rotation and priority-aware selection.
• Fair cycle and custom caps.
• Windowed quota tracking and quota groups.
• Evidence-based session tracking with compaction handling.
• Provider plugin discovery.
• Gemini CLI provider behavior unless a reviewed change is clearly better.
• Resilient file/JSON state writing.
• Dynamic OpenAI-compatible provider discovery.

Reference Gateway Ideas To Import Carefully

• Unified protocol/transformer style.
• Adapter registry and configurable provider/model adapters.
• Target groups and direct routing syntax, adapted into fallback-first routing.
• Responses API transformer and storage concepts.
• Stream TTFB/stall detection concepts, implemented with Python-native async primitives.
• Provider/model cooldown and retry-history concepts.
• Usage/cost normalization and provider-reported cost extraction.
• Broader provider support patterns for Claude Code, Codex, Copilot, and Antigravity.

Phase Index

1. Protocol Core.
2. Transform Pass Logging.
3. Adapter and Field Cache System.
4. Responses API and WebSocket-Ready Transport Shape.
5. Provider Protocol Overhaul.
6. Routing and Fallback Groups.
7. Retry/Cooldown/Failover Cleanup.
8. Streaming Library Upgrade.
9. Usage, Quota, and Cost Accuracy.
10. Config Polish.

Each phase may be subdivided if implementation scope becomes too large.

Completeness Matrix

This matrix exists so the branch does not lose any requested scope while phases evolve. The phase plans are still refreshed before implementation, but every item below must remain accounted for.

Requested area	Planned coverage
Protocols are priority #1	Phases 1 and 4 create native protocol foundations and Responses support before provider work.
Protocols are bases, not gospel	Phase 1 requires override-friendly protocol methods, subclassing, copy/mutate registration, and provider-specific overrides.
Move away from LiteLLM	Phase 1 adds a litellm_fallback protocol path; later providers should prefer native protocols and use LiteLLM only for unsupported coverage.
Add protocols automatically like providers	Phase 1 adds protocol auto-discovery and registry behavior modeled after provider discovery.
Cover current providers and reference providers	Phase 1 protocols must cover shapes used by current providers; Phase 5 covers Claude Code, Codex, Copilot, Antigravity, and Gemini CLI parity.
Responses API is very needed	Phase 4 is dedicated to Responses, previous_response_id, storage, SSE, and WebSocket-ready transport shape.
WebSocket support later	Phases 1, 4, and 8 require transport separation so WebSocket can be added without rewriting protocol logic.
Adapters/transformers tied to protocols	Phases 1, 2, and 3 define protocol parse/build plus transform tracing, adapter registry, and field-cache rules.
Cache and return provider fields	Phase 3 implements configurable extraction/injection rules for request, response, and stream fields with scope and mode controls.
Reasoning content and similar fields	Phase 3 explicitly covers reasoning content, thinking signatures, prompt cache keys, response IDs, and provider session IDs.
Return all possible or last user/assistant use	Phase 3 modes include last, all, last_user_turn, last_assistant_turn, and per_tool_call.
Per-model custom provider behavior	Phases 3, 5, and 10 cover provider/model field cache rules, adapters, model options, and optional JSON config.
Transaction logging after every transform	Phase 2 adds ordered request, response, and stream transform trace passes and integrates them with transaction logging.
Comments, docstrings, and key decisions	All implementation phases require docstrings for public abstractions and comments for non-obvious transform, protocol, and future-extension decisions.
Providers are priority #2	Phase 5 follows protocol foundations with Claude Code, Codex, Copilot, Antigravity, and Gemini CLI parity review.
Antigravity comparison	Phase 5 explicitly compares the reference Antigravity behavior against src/rotator_library/providers/_retired/.
Routing is interesting	Phase 6 implements fallback chains first, with target-group selectors later if useful.
Fallback groups preferred over target groups	Phase 6 starts with ordered fallback groups and only adds target-group-style selectors after that base works.
Retry/cooldown/failover cleanup	Phase 7 makes provider/model cooldown real, adds retry history, backoff, retry-after precedence, and success reset.
Quota/usage/cost improvements	Phase 9 adds protocol-aware normalizers, provider-reported cost extraction, structured cost fields, and checker abstractions while keeping existing usage engines.
Streaming as library capability	Phase 8 hardens streaming below the proxy route layer with TTFB, TTFT, stall detection, cancellation, and transport-aware stream events.
Config via env/json, no SQLite	Phase 10 adds optional JSON config with env overrides and validation. SQLite remains out of scope.
Multi-user proxy later	The branch keeps multi-user/admin features as a future expansion and only preserves extension points where natural.
Exhaustive tests in stages	Every phase requires tests alongside implementation and phase-end review by both explore and explore-heavy.
Reports are for the user, not git	06-phase-workflow.md says planning docs are committed, but phase reports are not committed by default.

Code Quality Expectations

• Public protocol, adapter, transport, field-cache, and provider-extension classes must have docstrings that explain intent, override points, and future expansion hooks.
• Non-obvious transformations must have comments explaining why data is changed, preserved, reordered, or intentionally dropped.
• Lossy protocol conversions must be documented at the conversion site.
• Future WebSocket, target-group, and multi-user extension seams should be noted in comments where they affect today's design.
• Tests should prefer golden fixtures for protocol shapes and focused unit tests for transform edge cases.

[mirrobot-agent]

Starting my review of the third-pass audit remediation changes — covering protocol output correctness, native operation guardrails, streaming hardening, Responses runtime improvements, cost accounting, routing parity, and much more across ~100 files. I'll analyze each subsystem and report back with a bundled review.

[mirrobot-agent]

Overall Assessment

This is a massive third-pass audit remediation that addresses findings from the explore and explore-heavy reviewers across all 10 phases of the experimental rewrite. The changeset spans ~9,000 lines across 100 files and is impressively well-scoped — each change maps back to a specific finding in third-pass-audit-findings.md.

The code quality is high. The streaming hardening (TTFB/stall timeouts, heartbeats, upstream close safety, CancelledError handling) is thorough and consistent across both the chat-completions StreamingHandler and the Responses stream_events path. The protocol output correctness fixes (OpenAI Chat/Responses usage formatting, legacy function_call handling, Ollama format_response, native operation guardrails) are all well-tested. The cost accounting improvements (provider-reported cost preservation, cache-write double-count fix, SSE cost comment parsing, structured cost breakdowns) are comprehensive and correctly prioritized.

Status of Previous Feedback

All 12 issues from the initial review and 5 suggestions from the follow-up have been addressed in this changeset:

• service.py CancelledError: Now caught by the Responses finally block and the Anthropic wrapper's finally block.
• Streaming adapter chain gap: The NativeProviderExecutor.stream() now runs stream-event adapter chains and extracts usage/cost.
• Bare except Exception: pass: Replaced with structured error handling and _can_retry_stream_after_error with explicit emitted_output tracking.
• Multi-turn tool use: _parent_output_to_messages now handles function_call and custom_tool_call items, and _message_to_chat handles tool role messages.
• Auth header mismatch: ClaudeCodeProvider now supports auto/x-api-key/bearer modes via CLAUDE_CODE_AUTH_HEADER.
• Hard litellm dependency: Not directly addressed in this diff, but the cost system now has provider-reported cost that bypasses litellm pricing — worth verifying this is tracked.

Key Suggestions

• Model-scoped cooldown bypasses failure-history threshold — contradicts Phase 7c plan (see inline)
• finally block recovery can raise — acquire_task.result() can replace the original exception (see inline)
• Polling logic duplication — next_upstream_chunk and acquire_upstream_stream share ~90% of their structure (see inline)
• Disconnect task leak — if is_disconnected() raises, the task is never cleaned up (see inline)
• Gemini cache-write double-count inconsistency — OpenAI path fixed but Gemini path not (see inline)
• Private _save_to_disk coupling — fragile dependency on provider cache internals (see inline)

Architectural Notes

The new streaming runtime settings system (get_stream_runtime_settings()) is well-designed — lazy-loaded via _retry_settings() to avoid import cycles, with env overriding JSON. The routing attempt history (_append_routing_attempt_history) is a solid observability addition that correctly sanitizes error types and tracks timing. The native provider cross-protocol formatting (client_protocol_name on NativeProviderContext) is a clean solution to the "responses formatted in provider protocol" blocker.

One architectural observation: the Responses streaming path now has its own independent polling/timeout/heartbeat implementation rather than reusing the StreamingHandler infrastructure. This is understandable given the different event model (Responses events vs SSE chunks), but it means streaming policy changes must be applied in two places.

This review was generated by an AI assistant.

[mirrobot-agent]

This hard early return for scope == "model" bypasses the failure-history threshold check entirely:

if scope == "model":
    return ProviderCooldownDecision(True, duration=duration, reason="model_capacity_cooldown", scope=scope, model=model, backoff_level=backoff_level)

This means a single server_error without retry_after will always start a model-scoped cooldown immediately, even if it's the first transient failure. The Phase 7c plan states: "No-retry_after server_error and api_connection should start cooldown only after FailureHistory.backoff_for() crosses the configured threshold." The provider-scoped path (below) correctly checks the threshold, but the model-scoped path does not.

If this is intentional (model-capacity errors are more actionable than generic transients), consider adding a comment explaining the policy difference. Otherwise, the model-scoped path should also check backoff_level > 0 before returning True.

Suggested change

	if error_type in {"server_error", "api_connection"} and default_duration >= provider_cooldown_min_seconds:
	backoff_level = 0
	duration = int(default_duration)
	if scope == "model":
	return ProviderCooldownDecision(True, duration=duration, reason="model_capacity_cooldown", scope=scope, model=model, backoff_level=backoff_level)
	if failure_history is None:
	return ProviderCooldownDecision(False, reason="missing_failure_history", scope=scope, model=model if scope == "model" else None)
	backoff = failure_history.backoff_for(provider=provider, error_type=error_type, scope=scope, model=model if scope == "model" else None, default_duration=duration)
	if scope == "model":
	if backoff_level <= 0 and failure_history is not None:
	return ProviderCooldownDecision(False, reason="transient_backoff_threshold_not_met", scope=scope, model=model)
	return ProviderCooldownDecision(True, duration=duration, reason="model_capacity_cooldown", scope=scope, model=model, backoff_level=backoff_level)

[mirrobot-agent]

This recovery path in the finally block can itself raise an exception, replacing the original one:

if chat_stream is None and acquire_task is not None and acquire_task.done() and not acquire_task.cancelled():
    chat_stream = acquire_task.result()  # Can raise!

If acquire_task completed with a ResponsesServiceError (e.g., from a TTFB timeout), calling .result() will re-raise that error inside the finally block, masking whatever exception triggered the finally in the first place.

Suggested change

	yield ResponsesStreamEvent("response.failed", failed)
	self._trace(transaction_logger, "stream_done_event", {"raw": "done"}, direction="stream", stage="final", metadata={"transport": transport, "failed": True})
	yield ResponsesStreamEvent("done", {}, terminal=True)
	finally:
	if chat_stream is None and acquire_task is not None and acquire_task.done() and not acquire_task.cancelled():
	try:
	chat_stream = acquire_task.result()
	stream_iterator = chat_stream.__aiter__()
	except Exception:
	chat_stream = None
	await cancel_task(pending_next_task)
	if chat_stream is None and acquire_task is not None and acquire_task.done() and not acquire_task.cancelled():
	try:
	chat_stream = acquire_task.result()
	stream_iterator = chat_stream.__aiter__()
	except Exception:
	chat_stream = None

[mirrobot-agent]

next_upstream_chunk (~90 lines) and acquire_upstream_stream (~55 lines) share nearly identical polling/disconnect/timeout/heartbeat logic. The structural pattern is the same: create task -> loop with asyncio.wait -> check disconnect -> check timeout -> emit heartbeat -> get result.

Any future change to the polling pattern (e.g., adding a new policy check) must be applied in both places. Consider extracting a shared _poll_upstream_async(task_factory, *, first: bool) helper that both call sites use, with the task factory being the only difference (stream_iterator.__anext__ vs client.acompletion).

[mirrobot-agent]

If request.is_disconnected() raises an unexpected exception, the disconnect_task created via asyncio.create_task(request.is_disconnected()) will never be awaited or cancelled. Python 3.9+ warns about unretrieved task exceptions, and Python 3.12+ logs Task was destroyed but it is pending! warnings.

Consider wrapping the disconnect task creation and cleanup in a try/finally pattern within the wait loop, or cancelling it explicitly in all exit paths (including the except Exception re-raise below).

Suggested change

	done, _ = await asyncio.wait(wait_tasks, timeout=wait_seconds)
	if disconnect_task is not None:
	if disconnect_task in done and disconnect_task.result():
	stream_cancelled = True
	next_task.cancel()
	with contextlib.suppress(asyncio.CancelledError, StopAsyncIteration):
	await next_task
	await close_upstream("client_disconnect")
	return
	if not disconnect_task.done():
	disconnect_task.cancel()
	with contextlib.suppress(asyncio.CancelledError):
	await disconnect_task
	if next_task in done:
	chunk = next_task.result()
	break
	timeout_error = _stream_timeout_error(monitor, stream_settings)
	if timeout_error:
	next_task.cancel()
	with contextlib.suppress(asyncio.CancelledError, StopAsyncIteration):
	await next_task
	await close_upstream(timeout_error[0], force=True)
	disconnect_task = None
	try:
	if request is not None:
	disconnect_task = asyncio.create_task(request.is_disconnected())
	wait_tasks.add(disconnect_task)
	done, _ = await asyncio.wait(wait_tasks, timeout=wait_seconds)
	if disconnect_task is not None:
	if disconnect_task in done and disconnect_task.result():
	stream_cancelled = True
	next_task.cancel()
	with contextlib.suppress(asyncio.CancelledError, StopAsyncIteration):
	await next_task
	await close_upstream("client_disconnect")
	return
	if not disconnect_task.done():
	disconnect_task.cancel()
	with contextlib.suppress(asyncio.CancelledError):
	await disconnect_task
	finally:
	if disconnect_task is not None and not disconnect_task.done():
	disconnect_task.cancel()
	with contextlib.suppress(asyncio.CancelledError):
	await disconnect_task

[mirrobot-agent]

The OpenAI-like path now correctly subtracts both cache_read and cache_write from prompt_tokens to avoid double-counting:

input_tokens = max(0, prompt_tokens - cache_read - cache_write)

However, _from_gemini_usage still only subtracts cache_read:

input_tokens=max(0, prompt_tokens - cache_read)

If a Gemini provider starts reporting cache_write_tokens (or cachedWriteContentTokens), those tokens would be double-counted in the normalized total. The Gemini protocol already parses cache_write_tokens from the response, so this is a latent consistency gap rather than a hypothetical.

[mirrobot-agent]

This directly accesses a private method on the provider cache:

flush = getattr(self._cache, "_save_to_disk", None)
if callable(flush):
    await flush()

If the provider cache implementation refactors _save_to_disk to a different name (e.g., _flush_to_disk, _persist), saves will silently stop flushing to disk, and the Responses store will appear to lose data between restarts.

Consider either: (a) adding a public flush() or sync() method to the cache interface that this code calls, or (b) adding a comment documenting the coupling so future maintainers know to update this call site.