docs: add user workflow + persona demo recordings

MLService/inference/embedder/requirements.txt

@@ -1,4 +1,4 @@
-fastapi==0.118.0
+fastapi==0.121.0
 uvicorn[standard]==0.32.1
 starlette==0.49.1
 h11==0.16.0

README.md

@@ -280,6 +280,38 @@ The full machine-readable trace lives in
 including timestamps for every frame, which is useful when diffing
 pipeline latency across releases.
 
+### 4.1 Full user workflow, current stack — 2026-07-25
+
+[`e2e-recordings/2026-07-25-user-workflow/videovault-user-workflow-recording.mp4`](e2e-recordings/2026-07-25-user-workflow/videovault-user-workflow-recording.mp4)
+re-runs the same kind of walkthrough against the current v2 stack (post
+Dependabot/OSV security fixes): real Google SSO → upload a 4K video →
+watch the live SSE pipeline through to **Ready** → search hits the
+confidence-threshold empty state → frame modal with inline playback and
+next-frame nav → a fresh `/ask` WebSocket chat session with a real streamed
+Qwen2.5-VL answer. Details, caveats (fallback captions, a build fix needed
+to bring the stack up) in
+[`e2e-recordings/2026-07-25-user-workflow/README.md`](e2e-recordings/2026-07-25-user-workflow/README.md).
+
+### 4.2 User persona / few-shot domain adaptation demo — 2026-07-29
+
+[`e2e-recordings/2026-07-29-user-persona-demo/videovault-user-persona-demo-recording-narrated.mp4`](e2e-recordings/2026-07-29-user-persona-demo/videovault-user-persona-demo-recording-narrated.mp4)
+(narrated, 79s) walks through VideoVault's **Domains** feature — the
+few-shot / user-persona mechanism — using the real CCTV Action Recognition
+dataset already indexed in the account. Same specific-action query
+(`"a person holding a gun"`, not a generic term) throughout:
+
+| Domain | State | Result |
+|---|---|---|
+| `General` | no persona | **0 results** |
+| `CCTV Crime Analyst` | freshly created, 1 label, 0 examples | **0 results** — a domain alone does nothing |
+| `CCTV adaptive demo` | trained: 3 labels, 23 examples (`gun · 9`) | **24/24 results**, top hit 88% confidence, `Action gun: 100%` |
+
+Closes with a live search-feedback thumbs-up: the domain's `Feedback`
+counter and model version update in real time (*"Saved. Future results
+adapt to this domain."*). Full writeup, including the file-upload
+constraint that shaped the approach, in
+[`e2e-recordings/2026-07-29-user-persona-demo/README.md`](e2e-recordings/2026-07-29-user-persona-demo/README.md).
+
 ---
 
 ## 5. Repo layout

e2e-recordings/2026-07-25-user-workflow/README.md

@@ -0,0 +1,63 @@
+# VideoVault User Workflow Recording — 2026-07-25
+
+Created: 2026-07-25T07:55:00Z
+
+Capture mode: Claude in Chrome `gif_creator` (real Chrome, real Google session) →
+GIF (50 frames, click indicators + action labels) → re-encoded to MP4 with ffmpeg.
+
+Stack: full local `docker compose` stack (web, api, mongo, postgres, redis,
+qdrant, embedder, chunker, normalizer, extractor) + host Ollama (`qwen2.5vl:7b`).
+
+## Flow covered
+
+- Signed-out landing page
+- Real Google OAuth2 SSO (completed manually in-browser — login/consent can't
+  be automated; see Notes)
+- Authenticated `/library` (v2 UI: Domain selector, min-confidence slider,
+  Labels/Features nav — this is the post-security-fix stack, not the June 16
+  recording's older UI)
+- Real multipart upload of `13258882-uhd_3840_2160_30fps.mp4` (86.4 MB, 4K)
+- Live SSE status through the full pipeline: queued → chunking → normalized →
+  indexing → **Ready (1/1 chunks indexed)**
+- Frame search UX: a strict query at the default 80% confidence threshold
+  returned zero results ("No frames passed the 80% confidence threshold"),
+  demonstrating the threshold-gating and "Lower confidence" / "Search general"
+  fallback affordances
+- Switched to the **"CCTV adaptive demo"** domain (a pre-trained domain with
+  3 labels / 23 examples) and re-ran the query, returning 6 high-confidence
+  frame hits
+- Frame modal: opened a hit, played the underlying clip inline, navigated to
+  the next neighboring frame (score panel updates: domain/visual/action/
+  feedback)
+- `/ask` — opened a fresh WebSocket chat session ("WebSocket connected"),
+  sent a new prompt about the just-uploaded video, and received a full
+  streamed RAG answer grounded in the retrieved frame captions
+
+## Notes
+
+- **Real Google SSO cannot be driven by the automation tooling** — the
+  browser tool refuses any interaction (including screenshots) on
+  `accounts.google.com`. The account-chooser/consent step was completed by
+  the user directly in the same Chrome tab; automation resumed once the
+  browser redirected back to `/library`.
+- **Frame captions are motion-heuristic fallbacks** ("mostly static cctv
+  video segment", "low motion ... video segment"), not full VLM scene
+  descriptions — same behavior as the June 16 recording. The `/ask` chat
+  answer, by contrast, is a real Qwen2.5-VL generation grounded in retrieval
+  context, so the VLM path itself works; only the per-frame captioning step
+  is degraded (likely hitting the extractor's soft-timeout fallback under
+  local Ollama latency).
+- Qdrant retained embeddings from a prior ingestion run (visible as the
+  `NTU_fight0160_gun_1.mp4` sourced hits) — the search results shown mix
+  frames from that older corpus with the newly uploaded video.
+- **Build fix required to bring the stack up**: `MLService/inference/embedder/requirements.txt`
+  pinned `fastapi==0.118.0` with `starlette==0.49.1`, which is outside
+  fastapi 0.118's `starlette<0.49.0` ceiling — pip's resolver failed the
+  `embedder` image build. Bumped to `fastapi==0.121.0` (ceiling
+  `starlette<0.50.0`) to resolve.
+- **Port 8080 conflict**: an unrelated `dataflow-control-plane` (kind/k8s)
+  container already held host port 8080. Stopped it (`docker stop
+  dataflow-control-plane`) to free the port for the `api` service, since
+  VideoVault's Google OAuth redirect URI is registered as
+  `http://localhost:8080/...` and can't be remapped without console access.
+  Restart it with `docker start dataflow-control-plane` when done.

e2e-recordings/2026-07-25-user-workflow/manifest.json

@@ -0,0 +1,31 @@
+{
+  "title": "VideoVault Real User Workflow Recording",
+  "createdAt": "2026-07-25T07:55:00Z",
+  "captureMode": "Claude in Chrome gif_creator (real browser session) -> GIF -> ffmpeg MP4",
+  "stack": "local docker compose (web, api, mongo, postgres, redis, qdrant, embedder, chunker, normalizer, extractor) + host Ollama qwen2.5vl:7b",
+  "uploadedVideo": "/Users/aryaman.sinha/Downloads/13258882-uhd_3840_2160_30fps.mp4",
+  "testedFlow": [
+    "Signed-out landing page",
+    "Real Google OAuth2 SSO (manual, in-browser)",
+    "Authenticated /library (v2 UI)",
+    "Multipart upload of 86.4MB 4K video",
+    "Live SSE pipeline status: queued -> chunking -> normalized -> indexing -> Ready",
+    "Search at default 80% confidence: zero results, threshold-gating UX",
+    "Domain switch to 'CCTV adaptive demo'",
+    "Search 'low motion': 6 high-confidence frame hits",
+    "Frame modal: inline clip playback + next-frame navigation with live re-score",
+    "/ask: fresh WebSocket session, new prompt about uploaded video, full streamed RAG answer"
+  ],
+  "buildFixes": [
+    {
+      "file": "MLService/inference/embedder/requirements.txt",
+      "issue": "fastapi==0.118.0 requires starlette<0.49.0 but starlette==0.49.1 was pinned, causing pip ResolutionImpossible",
+      "fix": "bumped fastapi to 0.121.0 (starlette<0.50.0 ceiling)"
+    }
+  ],
+  "environmentNotes": [
+    "Stopped unrelated dataflow-control-plane container to free host port 8080 (VideoVault api's Google OAuth redirect_uri is hardcoded to localhost:8080)",
+    "Qdrant retained embeddings from a prior ingestion run; search results include hits from that older corpus alongside the newly uploaded video",
+    "Frame captions are motion-heuristic fallbacks, not full VLM scene descriptions; /ask chat answers are real Qwen2.5-VL generations"
+  ]
+}

e2e-recordings/2026-07-29-user-persona-demo/README.md

@@ -0,0 +1,98 @@
+# VideoVault User Persona / Few-Shot Domain Demo — 2026-07-29
+
+Created: 2026-07-29T18:20:00Z
+
+Capture mode: Claude in Chrome `gif_creator` (real Chrome, real Google
+session) → GIF (50 frames, click indicators + action labels) → re-encoded to
+MP4 with ffmpeg.
+
+**Narrated version**:
+[`videovault-user-persona-demo-recording-narrated.mp4`](videovault-user-persona-demo-recording-narrated.mp4)
+(78.8s). Built as 6 independently-captured, independently-narrated segments
+rather than one voiceover laid over the raw recording — the first attempt at
+that (voiceover stretched to cover a single continuous capture) put the audio
+out of sync with what was on screen, and the raw capture's frame timing isn't
+proportional to real elapsed time in the first place. Instead, each of the 6
+milestones below was re-captured live as its own single-frame screenshot
+(app state is fully server-persisted and reproducible), narrated with its own
+`say -v Samantha` clip
+([`narration-script.txt`](narration-script.txt), one paragraph per segment),
+and each segment's still image is held on screen for exactly its narration's
+duration (`ffmpeg -loop 1 -t <narration_duration>`) before cutting to the
+next — guaranteeing narration and visual are aligned at every segment
+boundary, and guaranteeing the final segment shows the actual final state
+(not a frozen frame that happened to be the last one captured, which was
+wrong in the first attempt — it showed a stale/earlier domain).
+
+Real testing dataset used throughout: the
+[Kaggle CCTV Action Recognition dataset](https://www.kaggle.com/datasets/jonathannield/cctv-action-recognition-dataset)
+already ingested into this account's library in a prior benchmark session —
+2,300 real CCTV clips across 13 action classes (`fall`, `grab`, `gun`, `hit`,
+`kick`, `lying_down`, `run`, `sit`, `sneak`, `stand`, `struggle`, `throw`,
+`walk`). See `tests/bench/data/Videos/Videos/` and `tests/bench/queries.yaml`.
+
+## What this demonstrates
+
+**1. Normal (baseline) search on a specific instance query.** `Domain =
+General`, query `"a person holding a gun"` (a specific action, not a generic
+term like "low motion") → **zero results**, even after lowering the
+confidence threshold. Generic semantic search over this corpus's
+motion-heuristic fallback captions cannot find a specific action.
+
+**2. Few-shot domain / user persona — how it's built.** VideoVault's
+"Domains" feature (`/features`, "Manage learning" from the library) is the
+persona mechanism:
+- Created a fresh domain, `CCTV Crime Analyst` (name + context), to show the
+  onboarding flow from zero: *Create a domain → Add search labels → Upload
+  3-5 examples per label → Run searches and rate results*.
+- Added a label `gun` with model context `"a person holding or brandishing a
+  gun"` via the Few-shot labels panel. Domain state after this: `1 labels, 0
+  examples` — a domain with a label but no training signal.
+- Confirmed a brand-new domain with 0 examples performs **identically to
+  General** (also zero results for the same query) — creating a domain by
+  itself does nothing; it needs real few-shot signal.
+
+**3. The already-trained persona.** A domain from a prior session,
+`CCTV adaptive demo`, already has real few-shot training: **3 labels, 23
+examples** (`fall · 12`, `gun · 9`, `shooting · 1`), built from actual
+uploaded example clips + search-feedback (thumbs up/down), which the product
+describes as: *"Search feedback updates positive and negative centroids for
+this domain; examples provide the few-shot prototype signal."*
+
+**4. Persona search vs. normal search, same specific query.** `Domain = CCTV
+adaptive demo`, same query `"a person holding a gun"` → **24/24 results at
+≥80% confidence**, every hit tagged `Action gun`, top hit
+`NTU_fight0222_gun_1.mp4` at 86% confidence (`Visual 100%`, `Action 95%`) — a
+real CCTV clip of armed men. Zero → 24 for the identical query, General vs.
+the trained persona domain.
+
+**5. Adding a label live / making search more efficient.** Thumbs-upped a
+correct `gun` hit from the result list while `CCTV adaptive demo` was active.
+UI confirmed: *"Saved. Future results adapt to this domain."* Domain's
+`Feedback` counter moved `2 → 3` and the model version bumped
+(`v1782239 → v1785349`) — the live active-learning loop, in real time.
+
+## File upload constraint (why the new domain wasn't also trained with fresh uploads)
+
+The `Upload examples` few-shot flow requires a native file picker; automated
+`file_upload` is restricted to files the user explicitly shared with this
+session (chat attachments), not arbitrary paths on disk — including files
+already inside the project's own test dataset. Raw file-based few-shot
+training therefore has to be done by hand through the UI or via the
+`tests/bench/run_domain_adaptation_experiments.py` benchmark script (which
+authenticates via `BENCHMARK_AUTH_ENABLED` + `X-Benchmark-Google-Sub`, a
+*separate* synthetic account — its uploads don't appear in a real user's
+library). The **search-feedback path** (thumbs up/down on real results) has
+no such restriction and was used instead to demonstrate live label
+strengthening.
+
+## Notes
+
+- Frame captions across this corpus are still motion-heuristic fallbacks
+  ("mostly static", "low motion ... segment"), not full VLM scene
+  descriptions — same caveat as the `2026-07-25-user-workflow` recording.
+  The persona domain's `Action gun` / `Visual` / `Domain` score components
+  (not the caption text) are what actually carry the gun-specific signal.
+- Google SSO was completed manually by the user in-browser twice (session
+  expired mid-recording) — real automation cannot drive `accounts.google.com`
+  by design.

e2e-recordings/2026-07-29-user-persona-demo/manifest.json

@@ -0,0 +1,28 @@
+{
+  "title": "VideoVault User Persona / Few-Shot Domain Demo",
+  "createdAt": "2026-07-29T18:20:00Z",
+  "captureMode": "Claude in Chrome gif_creator (real browser session) -> GIF -> ffmpeg MP4",
+  "dataset": "Kaggle CCTV Action Recognition dataset (tests/bench/data/Videos/Videos), 2300 clips, 13 action classes",
+  "testedFlow": [
+    "Baseline: Domain=General, query 'a person holding a gun' -> 0 results even at lowered confidence",
+    "Created new domain 'CCTV Crime Analyst' via /features onboarding flow",
+    "Added few-shot label 'gun' with model context -> domain state: 1 labels, 0 examples",
+    "Confirmed untrained new domain performs identically to General (0 results) for same query",
+    "Switched to pre-trained domain 'CCTV adaptive demo' (3 labels, 23 examples: fall.12 gun.9 shooting.1)",
+    "Same query 'a person holding a gun' under trained persona -> 24/24 results >=80% confidence, all tagged Action gun",
+    "Top hit NTU_fight0222_gun_1.mp4 at 86% confidence (Visual 100%, Action 95%)",
+    "Live feedback: thumbs-up on a correct hit -> 'Saved. Future results adapt to this domain.', Feedback count 2->3, model version bumped"
+  ],
+  "comparison": {
+    "query": "a person holding a gun",
+    "general_domain_results": 0,
+    "untrained_new_domain_results": 0,
+    "trained_persona_domain_results": 24,
+    "trained_persona_top_confidence": "86%"
+  },
+  "constraints": [
+    "file_upload tool restricted to session-shared files only; could not upload fresh few-shot example clips from the local test dataset even though they're in the project directory",
+    "Used search-feedback (thumbs up/down) as the live few-shot signal instead, which has no such restriction",
+    "Real Google SSO completed manually by user in-browser (2x, session expired once mid-recording); accounts.google.com cannot be automated"
+  ]
+}

e2e-recordings/2026-07-29-user-persona-demo/narration-script.txt

@@ -0,0 +1,6 @@
+This is VideoVault. We start with a normal search under the General domain, asking for a very specific action: a person holding a gun. Zero results, even at the default threshold. Generic search struggles with a specific instance like this.
+So we build a user persona. On the Features page, we create a new domain, CCTV Crime Analyst, and add a few-shot label for gun, with a short description.
+But a brand new domain with zero examples behaves exactly like General: still zero results. Creating a domain alone isn't enough. It needs real training signal.
+So we switch to an already-trained persona, CCTV adaptive demo, built earlier from real few-shot examples, including nine for gun. Same exact query: twenty four out of twenty four results, all tagged Action gun.
+Here's the top hit, at eighty eight percent confidence. Domain match and Visual match both at one hundred percent, Action at ninety five percent. A real CCTV clip of armed men.
+Finally, the live feedback loop. A thumbs-up on a correct result saves instantly, adapting the domain in real time. Feedback count and model version update immediately. That's few-shot domain adaptation: zero results with generic search, twenty four relevant hits with a trained persona, and search that keeps getting better the more you label.