docs(examples): add rag-params-finder external project guide

examples/README.md

@@ -1,6 +1,10 @@
 # Examples
 
-A project gallery of full end-to-end applications built with SIE. Each project lives in its own subdirectory. Clone it, run it, learn from it.
+A project gallery of full end-to-end applications built with SIE. Most entries
+are self-contained under `examples/<name>/` — clone this repo, run them locally,
+and learn from them. Rows marked **External project guide** are docs-only
+landings that deep-link to a separately maintained repository (clone and run
+there).
 
 New to SIE? Start with the **[quickstart notebook](./quickstart.ipynb)** [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/superlinked/sie/blob/main/examples/quickstart.ipynb): encode, score, and extract in 5 minutes, then pick a project below.
 
@@ -9,7 +13,8 @@ New to SIE? Start with the **[quickstart notebook](./quickstart.ipynb)** [![Open
 Use this table to pick the right starting point. "Runnable" means the
 example has code, sample data or data-fetch instructions, and a documented
 local path. "Advanced" examples may require a custom SIE image or third-party
-service keys.
+service keys. "External project guide" means docs-only onboarding that deep-links
+to a separately maintained repository (clone and run there).
 
 | Example | Best for | SIE primitives | Setup | Status |
 |---|---|---|---|---|
@@ -33,6 +38,7 @@ service keys.
 | [Reconstruct a bearing failure](./maintenance-triage-agent) | Turning the NTSB's three East Palestine detector readings into a cited temperature and alert sequence without adding a new causal claim | `extract`, `encode`, `score` | SIE endpoint; standalone `uv` project; exact NTSB illustrated report spread | Runnable agent example |
 | [Make a shelf gap auditable](./retail-shelf-audit) | Detecting one empty facing, deriving its notice and shelf-label crops by geometry, then preserving OCR evidence | `extract` | GPU SIE deployment; standalone `uv` project; CC0 supermarket shelf image and recorded direct-checkpoint evidence included | Runnable evaluation example |
 | [A behavioural gate that catches hijacked AI agents by their actions, not their credentials](./agent-action-monitor) | Judging a proposed AI agent action against that agent's own learned baseline in real time, before it reaches a downstream system | `encode`, `score`, `extract` | Docker Compose (gate + self-hosted SIE + n8n + mock downstream), no API key required | Runnable demo |
+| [Find the best RAG config before you build](./rag-params-finder) | Sweeping embeddings × chunking × retrieval on your data before building a RAG app | `encode`, `score` (optional rerank) | External repo; MongoDB local or Atlas/Postgres; SIE gateway or Docker | External project guide |
 
 For docs publishing, lead with the quickest runnable demos, then use the
 benchmark and evaluation examples for deeper technical users.
@@ -41,14 +47,24 @@ benchmark and evaluation examples for deeper technical users.
 
 We welcome contributions. To add your project to the gallery:
 
+### Runnable examples (default)
+
 1. **Create a subdirectory** with a short, descriptive name (e.g. `wikipedia-search/`, `pdf-rag/`)
 2. **Include a README** that covers:
    - What the project does
    - How to run it (`docker compose up`, a script, etc.)
    - Which SIE features it uses (encode, score, extract, cluster, etc.)
-3. **Keep it self-contained** - include a `requirements.txt` or `package.json`, a docker-compose if needed, and sample data or instructions to fetch it
+3. **Keep it self-contained** — include a `requirements.txt` or `package.json`, a docker-compose if needed, and sample data or instructions to fetch it
 4. **Open a PR** against `main`
 
+### External project guides
+
+Use this path only when vendoring a runnable copy is impractical (large multi-service
+apps). Ship a thin `examples/<name>/` landing (README + short sibling pages) that
+deep-links to the external repo’s QUICKSTART/SIE setup, set Status to
+**External project guide**, and do **not** require in-tree `requirements.txt` /
+compose / sample data. See `examples/rag-params-finder/` for the shape.
+
 ### Review workflow
 
 Maintainers apply the `coderabbit-direct` label to eligible PRs that change content under `examples/**` or the root `README.md`. The label opts the PR into CodeRabbit review and allows CodeRabbit to formally approve it once review comments are resolved and required checks pass.

examples/rag-params-finder/README.md

@@ -0,0 +1,49 @@
+# Find the best RAG config before you build
+
+> Sweep embeddings × chunking × retrieval on **your** data — then ship the winning config first.
+
+This is an **external project guide**. The runnable app lives in
+[neomatrix369/rag-params-finder](https://github.com/neomatrix369/rag-params-finder)
+(MIT). This folder is the SIE-facing onboarding surface: short pages here, full
+detail in that repo.
+
+**SIE primitives used:** `encode` (embeddings via BGE-M3 / Stella-v5 / SPLADE-v3);
+optional `score` (SIE rerank). SIE is **opt-in** — the default stack runs without it.
+
+## Who this is for
+
+| You are… | Start here |
+|---|---|
+| New to SIE, found this in the gallery | [Getting started](./getting-started.md) → [SIE integration](./sie-integration.md) |
+| New to rag-params-finder, want SIE embeddings | Same path — then [What SIE does here](./what-sie-does.md) |
+
+Both audiences share the same **local** starting path (MongoDB stack +
+dashboard). SIE is optional afterward: configure a remote gateway or start
+self-hosted SIE, then run one `example-sie.yaml` sweep.
+
+## Start here
+
+1. [Getting started](./getting-started.md) — clone, prereqs, local Mongo path, dashboard
+2. [SIE integration](./sie-integration.md) — env vars, health checks, first SIE sweep
+3. [What SIE does here](./what-sie-does.md) — models, encode/score, vs Voyage/local
+4. [Troubleshooting](./troubleshooting.md) — short FAQ + deep-links
+
+**Canonical docs in the project:**
+[QUICKSTART](https://github.com/neomatrix369/rag-params-finder/blob/main/QUICKSTART.md) ·
+[SIE setup](https://github.com/neomatrix369/rag-params-finder/blob/main/docs/user-guide/sie-setup.md) ·
+[docs index](https://github.com/neomatrix369/rag-params-finder/blob/main/docs/README.md)
+
+## Ports cheat sheet
+
+| Service | Port | Notes |
+|---|---|---|
+| API | `8001` | rag-params-finder server |
+| Dashboard | `5374` | Experiments UI |
+| SIE (self-hosted only) | `8720` | Not started by `./start-services.sh` |
+
+## Attribution
+
+Built and maintained in
+[neomatrix369/rag-params-finder](https://github.com/neomatrix369/rag-params-finder)
+under the [MIT License](https://github.com/neomatrix369/rag-params-finder/blob/main/LICENSE).
+Screenshots and deeper guides live in that repository.

examples/rag-params-finder/getting-started.md

@@ -0,0 +1,60 @@
+# Getting started (without SIE yet)
+
+Goal: clone the external repo, start the local MongoDB stack, and open the
+dashboard. Enable SIE in [SIE integration](./sie-integration.md) after this works.
+
+**Source of truth:**
+[QUICKSTART.md](https://github.com/neomatrix369/rag-params-finder/blob/main/QUICKSTART.md)
+in the project repo.
+
+## Prerequisites
+
+- Git
+- Docker Desktop (running)
+- For host CLI later: Python 3.12+, [`uv`](https://docs.astral.sh/uv/)
+
+No Atlas account, Voyage key, or SIE credentials are required for this path.
+
+## Clone and start (MongoDB local)
+
+```bash
+git clone https://github.com/neomatrix369/rag-params-finder.git
+cd rag-params-finder
+cp .env.example .env
+./start-services.sh --mongodb-local
+```
+
+Open **http://localhost:5374**. The stack starts MongoDB Atlas Local, the API
+server (`:8001`), and the dashboard. SIE is **not** started — that is intentional.
+
+Verify the API:
+
+```bash
+curl -s http://localhost:8001/health
+```
+
+You should see the server healthy. With default `.env`, SIE reports as
+`"sie": "disabled"`.
+
+## Other storage paths
+
+| Path | When | Doc |
+|---|---|---|
+| Atlas cloud | You already have `MONGODB_URI` | [QUICKSTART Path B](https://github.com/neomatrix369/rag-params-finder/blob/main/QUICKSTART.md) |
+| Postgres / pgvector | Prefer Supabase or local Postgres | [Postgres setup](https://github.com/neomatrix369/rag-params-finder/blob/main/docs/user-guide/postgres-setup.md) |
+| Manual two-terminal | No Docker for the app | [QUICKSTART Path C](https://github.com/neomatrix369/rag-params-finder/blob/main/QUICKSTART.md) |
+
+Step-by-step install and first experiment:
+[getting-started.md](https://github.com/neomatrix369/rag-params-finder/blob/main/docs/user-guide/getting-started.md).
+
+## Next
+
+Before the host CLI / SIE handoff, export the URI the startup script printed
+(leave `.env` placeholders unchanged for Atlas Local):
+
+```bash
+# value also printed by ./start-services.sh --mongodb-local
+export MONGODB_URI="mongodb://localhost:27017/rag_params_finder?directConnection=true"
+```
+
+Then wire SIE and run one sweep: [SIE integration](./sie-integration.md).

examples/rag-params-finder/sie-integration.md

@@ -0,0 +1,150 @@
+# SIE integration
+
+SIE is **opt-in**. `./start-services.sh` and default Compose start the server +
+dashboard only — they never start SIE.
+
+**Source of truth:**
+[docs/user-guide/sie-setup.md](https://github.com/neomatrix369/rag-params-finder/blob/main/docs/user-guide/sie-setup.md).
+
+## Happy path — remote gateway (recommended)
+
+Finish [Getting started](./getting-started.md) first so `:8001` is up and
+`MONGODB_URI` is exported for the host CLI.
+
+In the project `.env`:
+
+```bash
+SIE_ENABLED=true
+SIE_ENDPOINT=https://your-sie-gateway.example.com
+SIE_API_KEY=your_gateway_token
+```
+
+Reload the server so it picks up the new env (a plain restart is not enough for
+Compose — env is baked in at container create time):
+
+```bash
+# Compose (typical after ./start-services.sh)
+docker compose up -d --force-recreate server
+
+# Host-run server instead: reload or restart uvicorn
+```
+
+Load only SIE vars into the **current shell** before gateway curls (do not
+`source .env` wholesale — that overwrites the host CLI `MONGODB_URI` export
+from [Getting started](./getting-started.md) with the Atlas placeholder):
+
+```bash
+export SIE_ENABLED=true
+export SIE_ENDPOINT=https://your-sie-gateway.example.com
+export SIE_API_KEY=your_gateway_token
+# keep the earlier MONGODB_URI export for Atlas Local host CLI
+```
+
+### Readiness checks
+
+**1. Gateway process alive** (`/healthz` ≠ model ready):
+
+```bash
+curl --connect-timeout 5 --max-time 15 \
+  -H "Authorization: Bearer $SIE_API_KEY" "$SIE_ENDPOINT/healthz"
+# → ok
+```
+
+**2. Model can encode** — accept only HTTP **200**; retry **503** (warm-up);
+stop on terminal failures (e.g. **502**, **401**):
+
+```bash
+attempts=0
+# 60 attempts × (up to 30s request + 10s sleep) ≈ 40 minutes worst case —
+# first-run model download/load can exceed a short 10-minute budget.
+max_attempts=60
+while true; do
+  attempts=$((attempts + 1))
+  code=$(curl --connect-timeout 5 --max-time 30 -s -o /dev/null -w '%{http_code}' \
+    -X POST "$SIE_ENDPOINT/v1/encode/BAAI/bge-m3" \
+    -H "Authorization: Bearer $SIE_API_KEY" \
+    -H "Content-Type: application/json" \
+    -d '{"items":[{"text":"readiness probe"}]}' || true)
+  case "$code" in
+    200) echo "SIE encode ready"; break ;;
+    503) echo "SIE warm-up ($attempts/$max_attempts) — waiting 10s..." ;;
+    000) echo "SIE unreachable ($attempts/$max_attempts) — waiting 10s..." ;;
+    *) echo "SIE encode failed with HTTP $code — abort"; exit 1 ;;
+  esac
+  if [ "$attempts" -ge "$max_attempts" ]; then
+    echo "SIE encode not ready after $max_attempts attempts — abort"
+    exit 1
+  fi
+  sleep 10
+done
+```
+
+**3. App sees SIE:**
+
+```bash
+curl -s http://localhost:8001/health
+# → "sie":"reachable"
+```
+
+### First success
+
+Provide an input PDF (`input_data/` is gitignored). Either copy a file to the
+path expected by the example config, or point `data_paths` at an existing PDF:
+
+```bash
+mkdir -p input_data/pdfs
+cp /path/to/your-document.pdf \
+  input_data/pdfs/The_Federal_Pell_Grant_Program.pdf
+# or edit data_paths in configs/mongodb/example-sie.yaml
+```
+
+Then run one sweep (CLI installed per project QUICKSTART):
+
+```bash
+rag-params-finder run --config configs/mongodb/example-sie.yaml
+```
+
+Config file:
+[configs/mongodb/example-sie.yaml](https://github.com/neomatrix369/rag-params-finder/blob/main/configs/mongodb/example-sie.yaml).
+On Postgres/Supabase use
+[configs/supabase/example-sie.yaml](https://github.com/neomatrix369/rag-params-finder/blob/main/configs/supabase/example-sie.yaml)
+instead.
+
+Compare results in the dashboard at **http://localhost:5374**.
+
+## Alternate — self-hosted Docker
+
+Use when you have no remote gateway. Needs Docker, disk for model weights, and
+**requires** `HF_TOKEN` on the **SIE container** for Hugging Face weight
+downloads during warm-up (not used for app routing).
+
+Typical host endpoint:
+
+```bash
+SIE_ENABLED=true
+SIE_ENDPOINT=http://localhost:8720
+# SIE_API_KEY usually unset for local unauthenticated server
+```
+
+Full `docker run` flags, warm-up (wait for encode **200**, not only `/healthz`),
+Apple Silicon notes, and Aim UI:
+[Self-hosted Docker in sie-setup.md](https://github.com/neomatrix369/rag-params-finder/blob/main/docs/user-guide/sie-setup.md).
+
+When the server runs in Docker and SIE on the host, use
+`http://host.docker.internal:8720` as `SIE_ENDPOINT`.
+
+## Env vars (same for remote and local)
+
+| Variable | Role |
+|---|---|
+| `SIE_ENABLED` | Master on/off (default `false`) |
+| `SIE_ENDPOINT` | HTTP base URL of the SIE gateway or local server |
+| `SIE_API_KEY` | Bearer token when the gateway requires auth |
+
+Details and smoke tests (including `POST /api/v1/sweep`):
+[sie-setup.md](https://github.com/neomatrix369/rag-params-finder/blob/main/docs/user-guide/sie-setup.md).
+
+## Next
+
+Understand models and primitives: [What SIE does here](./what-sie-does.md).
+Stuck? [Troubleshooting](./troubleshooting.md).

examples/rag-params-finder/troubleshooting.md

@@ -0,0 +1,48 @@
+# Troubleshooting (SIE-focused)
+
+Short FAQ for gallery readers. Full tables and recovery steps:
+[project troubleshooting](https://github.com/neomatrix369/rag-params-finder/blob/main/docs/user-guide/troubleshooting.md)
+(especially the SIE section) and
+[sie-setup.md](https://github.com/neomatrix369/rag-params-finder/blob/main/docs/user-guide/sie-setup.md).
+
+## `"sie": "disabled"` on `/health`
+
+`SIE_ENABLED` is false or unset (default). Set `SIE_ENABLED=true`, set
+`SIE_ENDPOINT`, reload the server.
+
+## `"sie": "unreachable"` (or preflight fails)
+
+- Wrong `SIE_ENDPOINT` (typo, http vs https, missing port)
+- Gateway auth: set `SIE_API_KEY` and use `Authorization: Bearer …` on `/healthz`
+- Local Docker: SIE not running, still warming up, or server-in-Docker needs
+  `http://host.docker.internal:8720`
+- Encode still returning **503** during model load — wait until encode returns **200**
+
+## Sweep with `provider: sie` fails immediately
+
+SIE guard runs preflight. Fix health/`SIE_ENABLED` first, then re-run. Index
+requirements come from the **selected config**, not from `provider: sie` alone.
+For [`configs/mongodb/example-sie.yaml`](https://github.com/neomatrix369/rag-params-finder/blob/main/configs/mongodb/example-sie.yaml)
+(dense BGE-M3 / Stella-v5), create both indexes on the `chunks` collection:
+`vector_index_1024` for dense 1024-dim embeddings, and `text_search_index` for
+the separately swept sparse/hybrid retrievers — see project MongoDB setup.
+Sparse-only models can need different indexes; do not treat `vector_index_1024`
+as universal.
+
+## `./start-services.sh` did not bring up SIE
+
+Expected. Start a remote gateway or follow self-hosted Docker in
+[sie-setup.md](https://github.com/neomatrix369/rag-params-finder/blob/main/docs/user-guide/sie-setup.md).
+
+## Dashboard up but no SIE experiments
+
+Use a SIE config such as
+[`configs/mongodb/example-sie.yaml`](https://github.com/neomatrix369/rag-params-finder/blob/main/configs/mongodb/example-sie.yaml),
+not a Voyage-only or local-only example.
+
+## Still stuck?
+
+1. [sie-setup.md](https://github.com/neomatrix369/rag-params-finder/blob/main/docs/user-guide/sie-setup.md) — known issues, warm-up, Aim UI  
+2. [troubleshooting.md](https://github.com/neomatrix369/rag-params-finder/blob/main/docs/user-guide/troubleshooting.md) — indexes, Docker, storage  
+3. Open an issue on
+   [neomatrix369/rag-params-finder](https://github.com/neomatrix369/rag-params-finder/issues)