Self-Healing LLM Gateway

A production-grade AI gateway for OpenAI-compatible APIs with intelligent provider failover, automatic recovery, circuit breaking, semantic caching, tenant-aware rate limiting, and enterprise observability.

Screenshots

Frontend console	Successful chat
Grafana dashboard	Chaos demo

Overview

LLM Gateway is a reliability focused proxy for chat completion traffic. It exposes an OpenAI compatible API while routing requests across multiple configured providers. The gateway stores custom provider credentials in Redis with encrypted API keys, tracks provider health in sliding windows, opens circuits when a provider degrades, and reroutes traffic to healthier providers.

The project also includes a Next.js console for saving providers, sending chat requests, injecting chaos, and opening the monitoring tools.

Core Capabilities

OpenAI compatible API	Clients can point chat completion traffic at the gateway with a base URL change.
Provider routing	LiteLLM normalizes calls across configured OpenAI compatible providers.
Encrypted provider storage	Custom providers are persisted in Redis and API keys are encrypted with Fernet.
Health tracking	Redis sliding windows track success rate, p50 latency, p95 latency, p99 latency, and error type.
Circuit breaking	Providers move through closed, open, and half open states based on health thresholds.
Retry queue	Deferrable requests can move into a Redis queue with backoff, jitter, and idempotency.
Semantic cache	Repeated prompts can be served from Redis using normalized prompt hashes.
Tenant controls	Per tenant rate limits, monthly budgets, request logging, and PII redaction are included.
Observability	Prometheus metrics and a provisioned Grafana dashboard show traffic, errors, latency, circuit state, queue depth, and cost.

Architecture

Project Layout

frontend	Next.js console for provider setup, chat testing, chaos actions, and monitoring links.
backend/app/main.py	FastAPI app with chat, provider admin, health, readiness, docs, and metrics routes.
backend/app/providers.py	Provider selection, LiteLLM calls, failover, hedging, queue handoff, and metrics.
backend/app/runtime_config.py	Redis provider store with encrypted API keys.
backend/app/health.py	Redis sliding window provider health tracker.
backend/app/circuit_breaker.py	Provider circuit breaker state machine.
backend/app/queue.py	Redis retry queue with idempotency protection.
backend/app/chaos.py	Admin chaos injection for synthetic provider latency and errors.
monitoring	Prometheus and Grafana provisioning.

Run Locally

From the project root, build the containers:

docker compose build

Start the stack:

docker compose up

Open these services:

Frontend console	http://localhost:3001
Backend API	http://localhost:8000
API docs	http://localhost:8000/docs
Grafana	http://localhost:3000
Prometheus	http://localhost:9090

To stop the stack:

docker compose down

Configuration

Create your local environment file:

copy .env.example .env

Set the values that apply to your machine:

LLM_GATEWAY_ADMIN_API_KEY=your_admin_key
LLM_GATEWAY_METRICS_API_KEY=your_metrics_key
ENCRYPTION_KEY=your_fernet_key
LLM_GATEWAY_REDIS_URL=redis://redis:6379/0
LLM_GATEWAY_ENABLE_DOCS=true

Never commit real API keys. The local .env file is ignored by Git.

Frontend Usage

Open the console:

http://localhost:3001

Enter the admin key from your environment. Then use the provider form to save a custom OpenAI compatible API.

Example provider values:

Provider name: logfare
Model: openai/deepseek_v4_flash
API base URL: https://logfare.ai/v1
API key: your_api_key
Request classes: classification
Priority: 1
Enabled: checked

After saving the provider, send a chat request from the chat panel. The browser talks to the gateway, and the gateway calls the provider.

Custom Inference APIs

Any API that follows the OpenAI chat completion shape can be used.

For a hosted provider:

API base URL: https://your_provider.example.com/v1
Model: openai/your_model
API key: your_api_key

For a local model server running on your computer while the backend runs in a container:

API base URL: http://host.docker.internal:11434/v1
Model: openai/your_model
API key: local

Use host.docker.internal because localhost inside the backend container points to the container itself.

Monitoring

Prometheus scrapes the gateway metrics endpoint. Grafana is provisioned with a dashboard for:

RPS by provider	Error rate
p95 latency	Circuit state timeline
Failover events	Queue depth
Cost per hour by tenant and feature	Provider health at a glance

Chaos Demo

Use the frontend console to trigger provider chaos:

1. Start the stack.
2. Open the frontend console.
3. Save or select a provider.
4. Send a healthy chat request.
5. Open Grafana.
6. Click the Chaos action for the provider.
7. Send more chat requests.
8. Watch errors rise, the circuit open, and traffic reroute.
9. Wait for recovery and confirm the provider closes again.

Health Checks

The backend exposes:

http://localhost:8000/health
http://localhost:8000/ready

The health route confirms the process is alive. The ready route confirms Redis is reachable and at least one provider route can be used.

Tests

Activate the virtual environment:

.\venv\Scripts\Activate.ps1

Run the test suite:

pytest backend\tests

The tests cover health tracking, circuit breaker cycles, deferrable queue behavior, semantic cache, tenant limits, PII redaction, runtime provider encryption, and production hardening.

Pytest cache is stored in:

.tmp/pytest_cache

Troubleshooting

The frontend shows 503

This usually means no provider is available. Confirm that a provider is saved, enabled, mapped to the request class, and not blocked by an open circuit.

Saved providers disappear

Do not delete Docker volumes if you want Redis data to stay. Use normal container restart commands for everyday development.

Local provider does not respond

When the backend runs in a container, use host.docker.internal instead of localhost for services running on your computer.

Admin actions return 403

Use the admin key configured in LLM_GATEWAY_ADMIN_API_KEY.

Security Notes

This project is suitable as a portfolio grade implementation and local demo. Before production use, add managed secret storage, deployment specific identity controls, persistent Redis backups, stricter network policy, CI checks, and a full operational runbook.

License

MIT License. See LICENSE.