feat(runtime): add strict secrets mode to secrets provider

test(runtime): add strict-mode tests for file provider

docs: clarify async LLM in README and architecture; add TDS parser support matrix; document SECRET_PROVIDER_MODE; TDS scope/risk in ENFORCEMENT

chore(lint): fix E402/E702 and run fmt; translate Swedish comments in workflow

Author: Caripson

.env.example

@@ -28,6 +28,12 @@ LLM_ENDPOINT=http://localhost:11434
 LLM_SEND_EXTERNAL=false
 REDACTION_ENABLED=true
 
+# --- Secrets provider ---
+# Choose provider: env|file
+SECRET_PROVIDER=env
+# Behavior: permissive (fallback to env) | strict (error if file missing)
+SECRET_PROVIDER_MODE=permissive
+
 # --- Testing/integration toggles ---
 # When set, integration tests may connect to a live SQL Server
 ENABLE_INTEGRATION_TESTS=false

.github/workflows/publish-wiki.yml

@@ -22,14 +22,13 @@ jobs:
           cp docs/*.md wiki_temp/
           cd wiki_temp
 
-          # Byt namn på index.md till Home.md för att skapa startsidan
+          # Rename index.md to Home.md to create the Wiki landing page
           if [ -f "index.md" ]; then
             mv index.md Home.md
           fi
 
-          # *** HÄR ÄR DEN SISTA FIXEN ***
-          # Hitta alla .md-filer och ta bort ".md" från slutet av alla länkar inuti dem.
-          # Detta gör att länkarna fungerar korrekt i GitHubs Wiki-system.
+          # Final fix: remove trailing ".md" from internal links in all Markdown files
+          # so links work correctly in GitHub Wiki.
           find . -type f -name "*.md" -exec sed -i 's/\.md)/)/g' {} +
 
           git config user.name "github-actions[bot]"

README.md

@@ -18,13 +18,13 @@
     <br/>
 </p>
 
-SQLumAI is an invisible, AI‑powered proxy for Microsoft SQL Server.
+SQLumAI is an invisible proxy for Microsoft SQL Server with out‑of‑band AI insights.
 
 For non‑technical readers
 - What it does: Watches data flowing to SQL Server and helps improve data quality – without slowing anything down.
 - How it helps: Finds missing values, inconsistent formats (dates, phone numbers), and process gaps; proposes fixes and simpler input rules; summarizes issues daily.
 - Why it’s safe: It forwards traffic transparently by default (dry‑run). You control when to enforce rules.
-- Where AI fits: A local LLM turns raw events into a short list of high‑value actions and insights.
+- Where AI fits: A local LLM turns raw events into a short list of high‑value actions and insights (generated asynchronously, not inline).
 
 ### Dependencies
 - ![FastAPI](https://img.shields.io/pypi/v/fastapi?label=fastapi) ![Uvicorn](https://img.shields.io/pypi/v/uvicorn?label=uvicorn) ![Pydantic](https://img.shields.io/pypi/v/pydantic?label=pydantic) ![HTTPX](https://img.shields.io/pypi/v/httpx?label=httpx) ![prometheus_client](https://img.shields.io/pypi/v/prometheus_client?label=prometheus_client)
@@ -40,7 +40,7 @@ For non‑technical readers
 - MVP 3 – Gatekeeper: Rules API + engine, env‑gating, thresholds; optional TLS termination + TDS parsing for SQL Batch/RPC; simple column‑level autocorrect; metrics, audits, dashboards.
   - Dry‑run vs enforce: `ENFORCEMENT_MODE=log|enforce`
   - Parsers: `ENABLE_TDS_PARSER=true`, `ENABLE_SQL_TEXT_SNIFF=true`
-  - LLM: `LLM_PROVIDER`, `LLM_MODEL`, `LLM_ENDPOINT` (Ollama default), `OPENAI_API_KEY` (OpenAI-compatible)
+  - LLM: `LLM_PROVIDER`, `LLM_MODEL`, `LLM_ENDPOINT` (Ollama default), `OPENAI_API_KEY` (OpenAI-compatible). LLM runs async via scheduler jobs.
   - Scheduler: `ENABLE_SCHEDULER`, `SCHEDULE_INTERVAL_SEC`
   - Docs overview: see `docs/mvp.md`
 
@@ -52,7 +52,7 @@ For non‑technical readers
   - NL→Rule suggestion: `POST /rules/suggest` returns a proposed rule JSON from plain text.
   - XEvents helper API: `POST /xevents/setup?mode=ring|file` returns a ready-to-run SQL session script.
   - XEvents setup helper: `scripts/setup_xevents.py` renders a session SQL for ring buffer or file targets.
-  - Secrets provider: `src/runtime/secrets.py` with `SECRET_PROVIDER=env|file`.
+  - Secrets provider: `src/runtime/secrets.py` with `SECRET_PROVIDER=env|file` and `SECRET_PROVIDER_MODE=permissive|strict`.
 
 See `AGENTS.md` for contributor guidelines and development conventions.
 
@@ -81,9 +81,18 @@ flowchart LR
   API --> P
 ```
 
+### Real‑time vs Async AI
+- Real‑time: The proxy applies deterministic rules to decide allow/autocorrect/block. No LLM calls occur on the hot path.
+- Async: Scheduler jobs read aggregated data and generate insights and proposed rules with an LLM.
+- Safety: Local‑first by default; no external LLM traffic unless explicitly configured.
+
+### TDS Parser Support
+- Minimal, best‑effort parsing of Batch and RPC to keep the hot path safe.
+- See `docs/ENFORCEMENT.md` for current scope, limitations, and roadmap.
+
 Docs
 - Browse docs in `docs/` or serve with `mkdocs serve`.
-- MVPs: `docs/mvp.md`  |  Enforcement: `docs/ENFORCEMENT.md`  |  Architecture: `docs/architecture.md` | HA: `docs/ha.md`
+- MVPs: `docs/mvp.md`  |  Enforcement: `docs/ENFORCEMENT.md`  |  TDS Support: `docs/tds-parser.md`  |  Architecture: `docs/architecture.md` | HA: `docs/ha.md`
 - LLM config/providers: `docs/llm-providers.md`  |  Insights: `docs/insights.md`
 - Reports/Integration: `docs/howto-reports.md`, `docs/howto-integration.md`
 - Metrics dashboard: `docs/metrics-dashboard.md`
@@ -177,7 +186,7 @@ curl -s -X POST http://localhost:8080/rules \
 # Suggest a rule from natural language (stub)
 curl -s -X POST http://localhost:8080/rules/suggest \
   -H 'Content-Type: application/json' \
-  -d '{"text":"Alla svenska telefonnummer ska normaliseras"}' | jq .
+  -d '{"text":"Normalize all Swedish phone numbers"}' | jq .
 ```
 
 ## Usage Scenarios: BSS, Booking, ServiceNow, CRM
@@ -210,7 +219,7 @@ Principles:
 [
   {"id":"bss-phone-autocorrect","target":"column","selector":"dbo.Customers.Phone","action":"autocorrect","reason":"Normalize E.164","confidence":0.9},
   {"id":"bss-email-required","target":"column","selector":"dbo.Customers.Email","action":"block","reason":"Email required at onboarding","confidence":1.0},
-  {"id":"bss-no-test-subs","target":"pattern","selector":"INSERT INTO dbo.Subscriptions","action":"block","reason":"Stoppa testabonnemang i prod","confidence":0.9}
+  {"id":"bss-no-test-subs","target":"pattern","selector":"INSERT INTO dbo.Subscriptions","action":"block","reason":"Block test subscriptions in production","confidence":0.9}
 ]
 ```
 - LLM insights (examples):
@@ -225,7 +234,7 @@ Principles:
 ```json
 [
   {"id":"booking-no-overlap","target":"pattern","selector":"INSERT INTO dbo.Bookings","action":"block","reason":"Overlapping times must be prevented in app logic","confidence":0.8},
-  {"id":"booking-email-format","target":"column","selector":"dbo.Bookings.Email","action":"autocorrect","reason":"Korrigera vanliga typos","confidence":0.7}
+  {"id":"booking-email-format","target":"column","selector":"dbo.Bookings.Email","action":"autocorrect","reason":"Correct common typos","confidence":0.7}
 ]
 ```
 - LLM insights (examples):

docs/ENFORCEMENT.md

@@ -19,14 +19,14 @@ This document explains how SQLumAI evolves from passive monitoring to selective,
 - Policy engine: For each candidate value, evaluate rules in order; emit decision (allow/autocorrect/block) + reason + confidence.
 
 ### SQL Batch (0x01)
-- Reassembly of full batch, decode as UTF‑16LE; apply pattern/table rules.
-- Column‑level mapping for simple INSERT/UPDATE via regex parser; safe rewrite of SQL text when `ENFORCEMENT_MODE=enforce`.
-- Multirow INSERT stöd: omskrivning av `(…), (…)` block när kolumn‑värde‑antal matchar.
+- Reassemble full batch, decode as UTF‑16LE; apply pattern/table rules.
+- Column‑level mapping for simple INSERT/UPDATE via a best‑effort regex parser; safe SQL text rewrite when `ENFORCEMENT_MODE=enforce`.
+- Multirow INSERT support: rewrite `(…), (…)` groups when the number of columns matches the number of values.
 
 ### RPC (0x03)
-- Reassembly av RPC payload; heuristisk extraktion av NVARCHAR‑parametrar.
-- In‑place autocorrect (utf‑16le) när nya värdet inte är längre än det gamla (kortare pad: spaces) när `RPC_AUTOCORRECT_INPLACE=true`.
-- Blockering av RPC vid regelmatchning när `ENFORCEMENT_MODE=enforce`.
+- Reassemble RPC payload; heuristic extraction of NVARCHAR parameters.
+- In‑place autocorrect (utf‑16le) when the new value is not longer than the original (shorter padded with spaces) when `RPC_AUTOCORRECT_INPLACE=true`.
+- Block RPC on rule match when `ENFORCEMENT_MODE=enforce`.
 
 ## Traceability & Safety
 - Logging: Every correction/block records rule id, reason, confidence, original and resulting value.
@@ -40,8 +40,14 @@ This document explains how SQLumAI evolves from passive monitoring to selective,
 4) Monitor: Track metrics (auto‑corrections, blocks, false positives) to tune thresholds.
 
 ## Feature flags / Env gating
-- Per‑regel: `enabled: true/false`, `apply_in_envs: ["dev","staging","prod"]` för att styra var regler gäller.
-- Globala toggles: `ENABLE_TDS_PARSER`, `ENABLE_SQL_TEXT_SNIFF`, `ENFORCEMENT_MODE`, `RPC_AUTOCORRECT_INPLACE`, `TIME_BUDGET_MS`, `MAX_REWRITE_BYTES`.
+- Per‑rule controls: `enabled: true/false`, `apply_in_envs: ["dev","staging","prod"]` to limit where rules apply.
+- Global toggles: `ENABLE_TDS_PARSER`, `ENABLE_SQL_TEXT_SNIFF`, `ENFORCEMENT_MODE`, `RPC_AUTOCORRECT_INPLACE`, `TIME_BUDGET_MS`, `MAX_REWRITE_BYTES`.
+
+## TDS Parser Scope and Risk
+- SQL Server’s TDS protocol is complex. SQLumAI’s parsing is intentionally minimal and best‑effort to keep the hot path safe.
+- Supported today: UTF‑16LE batch text for simple INSERT/UPDATE matching and heuristic NVARCHAR RPC parameter extraction.
+- Not guaranteed: unusual datatypes, vendor‑specific RPCs, newer protocol nuances. On parse failure, the proxy fails open and logs context.
+- Roadmap: consider contributing to or integrating a robust external TDS library. Contributions welcome.
 
 ## Performance
 - Keep parsing minimal (batch/statement only); avoid heavy transforms on the hot path.

docs/architecture.md

@@ -23,3 +23,7 @@ flowchart LR
   XE --> R
   API --> P
 ```
+
+Notes
+- Real‑time decisions in the proxy are deterministic (rules engine). No LLM calls happen on the hot path.
+- LLM processing is asynchronous: scheduler jobs read aggregated data and generate insights and proposed rules.

docs/tds-parser.md

@@ -0,0 +1,56 @@
+# TDS Parser Support Matrix
+
+This document outlines what the built‑in TDS handling currently supports, what is explicitly out of scope, and the safety posture. The goal is transparency so operators can decide when to enable parsing and enforcement.
+
+## Philosophy
+- Keep the hot path safe and fast: minimal, best‑effort parsing only where it adds value.
+- Fail open: if parsing or mapping fails, the proxy forwards traffic unchanged and logs context.
+- Prefer explicit rules over heavy rewrites; keep autocorrects reversible and auditable.
+
+## Summary Matrix
+
+| Area | Support | Notes |
+|------|---------|-------|
+| TDS packet headers | Basic | Used for flow control and identifying packet types. |
+| SQL Batch (0x01) reassembly | Yes | UTF‑16LE decoding to recover batch text. |
+| SQL text analysis | Limited | Best‑effort regex for simple INSERT/UPDATE detection. |
+| Column mapping (INSERT) | Limited | Match column list to VALUES tuples when counts align. |
+| Multi‑row INSERT | Limited | Rewrites supported only when column/value counts match per tuple. |
+| Column mapping (UPDATE) | Limited | Heuristic mapping of SET column=value pairs (simple cases). |
+| MERGE/BULK/CTE/complex SQL | No | Not parsed beyond basic pattern checks; no rewrites. |
+| RPC (0x03) reassembly | Yes | Reconstruct payload for parameter extraction. |
+| RPC parameter types | Partial | Heuristic NVARCHAR extraction; other types not guaranteed. |
+| In‑place RPC autocorrect | Optional | When `RPC_AUTOCORRECT_INPLACE=true` and new value is not longer. |
+| TLS termination | Optional | Off by default; required to read payloads on the proxy. |
+
+## Supported Scenarios (Examples)
+- Detect and optionally rewrite trivial `INSERT INTO dbo.Table (A,B) VALUES ("x","y")` when rules target specific columns.
+- Heuristically extract NVARCHAR RPC parameters for rule checks and lightweight autocorrect when safe in place.
+
+## Not Covered / Limitations
+- Unusual or newer SQL Server datatypes (e.g., SQL_VARIANT, XML, TVP) are not guaranteed.
+- Vendor‑specific RPCs or non‑standard encodings.
+- Complex SQL constructs (MERGE, CTEs, nested queries) — analysis is pattern‑level only; no rewrites.
+
+## Safety & Failure Modes
+- Fail‑open by default: undecided/failed parsing → forward unchanged and log.
+- Bounded rewrites: controlled by `TIME_BUDGET_MS` and `MAX_REWRITE_BYTES`.
+- Auditable: all corrections/blocks include rule id, reason, and confidence in logs/metrics.
+
+## Configuration
+- Feature toggles: `ENABLE_TDS_PARSER`, `ENABLE_SQL_TEXT_SNIFF`, `ENFORCEMENT_MODE`, `RPC_AUTOCORRECT_INPLACE`, `TIME_BUDGET_MS`, `MAX_REWRITE_BYTES`.
+- TLS termination: `TLS_TERMINATION`, `TLS_CERT_PATH`, `TLS_KEY_PATH`.
+
+## Tests & Coverage
+- Unit tests for batch/RPC parsing heuristics live under `tests/` (e.g., `test_tds_parser.py`, `test_rpc_parse.py`).
+- Integration tests are gated by environment variables and are optional (`ENABLE_INTEGRATION_TESTS`).
+
+## Roadmap: External TDS Library
+To reduce protocol risk and broaden support, consider adopting or contributing to a robust TDS implementation. Evaluation criteria:
+- Protocol coverage (packets, datatypes, RPC formats, TLS).
+- Performance characteristics and memory footprint suitable for a proxy.
+- License compatibility (MIT‑friendly) and maintenance activity.
+- Extensibility hooks for safely mapping parameters to columns.
+
+If you have experience with suitable libraries or are interested in collaborating, please open an issue or PR.
+

scripts/replay_dryrun.py

@@ -10,7 +10,7 @@
 import datetime as dt
 from pathlib import Path
 from collections import defaultdict, Counter
-from src.policy.engine import PolicyEngine, Event, Rule
+from src.policy.engine import PolicyEngine, Event
 from src.policy.loader import load_rules
 from src.metrics import store as metrics_store
 

scripts/validate_rules.py

@@ -7,7 +7,7 @@
 import json
 import sys
 from pathlib import Path
-from typing import Any, Tuple
+from typing import Tuple
 
 
 def main():

src/api.py

@@ -1,54 +1,54 @@
 try:
     from fastapi import FastAPI, HTTPException, Response
-except Exception:  # minimal shim for environments without fastapi
-    class HTTPException(Exception):
-        def __init__(self, status_code: int = 500, detail: str = ""):
-            super().__init__(detail)
-            self.status_code = status_code
-
-    class Response:  # type: ignore
-        def __init__(self, content: str | bytes = b"", media_type: str = "text/html"):
-            self.content = content
-            self.media_type = media_type
-            # For tests accessing body
-            try:
-                self.body = content if isinstance(content, bytes) else str(content).encode("utf-8")
-            except Exception:
-                self.body = b""
-
-    class DummyApp:
-        def __init__(self, *_, **__):
-            pass
-
-        def get(self, *_args, **_kwargs):
-            def deco(fn):
-                return fn
-            return deco
-
-        def post(self, *_args, **_kwargs):
-            def deco(fn):
-                return fn
-            return deco
-
-        def delete(self, *_args, **_kwargs):
-            def deco(fn):
-                return fn
-            return deco
-
-    FastAPI = DummyApp  # type: ignore
+except Exception:  # minimal shim for environments without fastapi  # pragma: no cover
+    class HTTPException(Exception):  # pragma: no cover
+        def __init__(self, status_code: int = 500, detail: str = ""):  # pragma: no cover
+            super().__init__(detail)  # pragma: no cover
+            self.status_code = status_code  # pragma: no cover
+
+    class Response:  # type: ignore  # pragma: no cover
+        def __init__(self, content: str | bytes = b"", media_type: str = "text/html"):  # pragma: no cover
+            self.content = content  # pragma: no cover
+            self.media_type = media_type  # pragma: no cover
+            # For tests accessing body  # pragma: no cover
+            try:  # pragma: no cover
+                self.body = content if isinstance(content, bytes) else str(content).encode("utf-8")  # pragma: no cover
+            except Exception:  # pragma: no cover
+                self.body = b""  # pragma: no cover
+
+    class DummyApp:  # pragma: no cover
+        def __init__(self, *_, **__):  # pragma: no cover
+            pass  # pragma: no cover
+
+        def get(self, *_args, **_kwargs):  # pragma: no cover
+            def deco(fn):  # pragma: no cover
+                return fn  # pragma: no cover
+            return deco  # pragma: no cover
+
+        def post(self, *_args, **_kwargs):  # pragma: no cover
+            def deco(fn):  # pragma: no cover
+                return fn  # pragma: no cover
+            return deco  # pragma: no cover
+
+        def delete(self, *_args, **_kwargs):  # pragma: no cover
+            def deco(fn):  # pragma: no cover
+                return fn  # pragma: no cover
+            return deco  # pragma: no cover
+
+    FastAPI = DummyApp  # type: ignore  # pragma: no cover
 try:
     from pydantic import BaseModel, Field
-except Exception:  # minimal shim for environments without pydantic
-    class BaseModel:  # type: ignore
-        def __init__(self, **data):
-            for k, v in data.items():
-                setattr(self, k, v)
+except Exception:  # minimal shim for environments without pydantic  # pragma: no cover
+    class BaseModel:  # type: ignore  # pragma: no cover
+        def __init__(self, **data):  # pragma: no cover
+            for k, v in data.items():  # pragma: no cover
+                setattr(self, k, v)  # pragma: no cover
 
-        def model_dump(self):
-            return {k: getattr(self, k) for k in self.__dict__.keys()}
+        def model_dump(self):  # pragma: no cover
+            return {k: getattr(self, k) for k in self.__dict__.keys()}  # pragma: no cover
 
-    def Field(default=None, **_):  # type: ignore
-        return default
+    def Field(default=None, **_):  # type: ignore  # pragma: no cover
+        return default  # pragma: no cover
 from typing import List, Literal, Optional
 import json
 import os
@@ -59,8 +59,8 @@ def Field(default=None, **_):  # type: ignore
 from scripts.setup_xevents import render_xevents_sql
 try:
     from src.version import __version__
-except Exception:
-    __version__ = "0.0.0"
+except Exception:  # pragma: no cover
+    __version__ = "0.0.0"  # pragma: no cover
 
 app = FastAPI(title="SQLumAI Policy API", version=__version__)
 

src/proxy/tds_proxy.py

@@ -2,6 +2,7 @@
 import logging
 import os
 import time
+import contextlib
 from src.policy.loader import load_rules
 from src.policy.engine import PolicyEngine, Event
 from src.metrics import store as metrics_store
@@ -316,9 +317,6 @@ async def handle_client(local_reader: asyncio.StreamReader, local_writer: asynci
     logger.info(f"{conn_id} closed bytes c2s={counter.get('c2s',0)} s2c={counter.get('s2c',0)}")
 
 
-import contextlib
-
-
 async def run_proxy(listen_host: str, listen_port: int, upstream_host: str, upstream_port: int, stop_event: Optional[asyncio.Event] = None):
     server = await asyncio.start_server(
         lambda r, w: handle_client(r, w, upstream_host, upstream_port, f"conn-{id(w)}"), listen_host, listen_port