diff --git a/MLService/inference/embedder/requirements.txt b/MLService/inference/embedder/requirements.txt index 6db1f72..5026c90 100644 --- a/MLService/inference/embedder/requirements.txt +++ b/MLService/inference/embedder/requirements.txt @@ -1,4 +1,4 @@ -fastapi==0.118.0 +fastapi==0.121.0 uvicorn[standard]==0.32.1 starlette==0.49.1 h11==0.16.0 diff --git a/README.md b/README.md index 831759e..f0af32c 100644 --- a/README.md +++ b/README.md @@ -280,6 +280,38 @@ The full machine-readable trace lives in including timestamps for every frame, which is useful when diffing pipeline latency across releases. +### 4.1 Full user workflow, current stack — 2026-07-25 + +[`e2e-recordings/2026-07-25-user-workflow/videovault-user-workflow-recording.mp4`](e2e-recordings/2026-07-25-user-workflow/videovault-user-workflow-recording.mp4) +re-runs the same kind of walkthrough against the current v2 stack (post +Dependabot/OSV security fixes): real Google SSO → upload a 4K video → +watch the live SSE pipeline through to **Ready** → search hits the +confidence-threshold empty state → frame modal with inline playback and +next-frame nav → a fresh `/ask` WebSocket chat session with a real streamed +Qwen2.5-VL answer. Details, caveats (fallback captions, a build fix needed +to bring the stack up) in +[`e2e-recordings/2026-07-25-user-workflow/README.md`](e2e-recordings/2026-07-25-user-workflow/README.md). + +### 4.2 User persona / few-shot domain adaptation demo — 2026-07-29 + +[`e2e-recordings/2026-07-29-user-persona-demo/videovault-user-persona-demo-recording-narrated.mp4`](e2e-recordings/2026-07-29-user-persona-demo/videovault-user-persona-demo-recording-narrated.mp4) +(narrated, 79s) walks through VideoVault's **Domains** feature — the +few-shot / user-persona mechanism — using the real CCTV Action Recognition +dataset already indexed in the account. Same specific-action query +(`"a person holding a gun"`, not a generic term) throughout: + +| Domain | State | Result | +|---|---|---| +| `General` | no persona | **0 results** | +| `CCTV Crime Analyst` | freshly created, 1 label, 0 examples | **0 results** — a domain alone does nothing | +| `CCTV adaptive demo` | trained: 3 labels, 23 examples (`gun · 9`) | **24/24 results**, top hit 88% confidence, `Action gun: 100%` | + +Closes with a live search-feedback thumbs-up: the domain's `Feedback` +counter and model version update in real time (*"Saved. Future results +adapt to this domain."*). Full writeup, including the file-upload +constraint that shaped the approach, in +[`e2e-recordings/2026-07-29-user-persona-demo/README.md`](e2e-recordings/2026-07-29-user-persona-demo/README.md). + --- ## 5. Repo layout diff --git a/e2e-recordings/2026-07-25-user-workflow/README.md b/e2e-recordings/2026-07-25-user-workflow/README.md new file mode 100644 index 0000000..509520a --- /dev/null +++ b/e2e-recordings/2026-07-25-user-workflow/README.md @@ -0,0 +1,63 @@ +# VideoVault User Workflow Recording — 2026-07-25 + +Created: 2026-07-25T07:55:00Z + +Capture mode: Claude in Chrome `gif_creator` (real Chrome, real Google session) → +GIF (50 frames, click indicators + action labels) → re-encoded to MP4 with ffmpeg. + +Stack: full local `docker compose` stack (web, api, mongo, postgres, redis, +qdrant, embedder, chunker, normalizer, extractor) + host Ollama (`qwen2.5vl:7b`). + +## Flow covered + +- Signed-out landing page +- Real Google OAuth2 SSO (completed manually in-browser — login/consent can't + be automated; see Notes) +- Authenticated `/library` (v2 UI: Domain selector, min-confidence slider, + Labels/Features nav — this is the post-security-fix stack, not the June 16 + recording's older UI) +- Real multipart upload of `13258882-uhd_3840_2160_30fps.mp4` (86.4 MB, 4K) +- Live SSE status through the full pipeline: queued → chunking → normalized → + indexing → **Ready (1/1 chunks indexed)** +- Frame search UX: a strict query at the default 80% confidence threshold + returned zero results ("No frames passed the 80% confidence threshold"), + demonstrating the threshold-gating and "Lower confidence" / "Search general" + fallback affordances +- Switched to the **"CCTV adaptive demo"** domain (a pre-trained domain with + 3 labels / 23 examples) and re-ran the query, returning 6 high-confidence + frame hits +- Frame modal: opened a hit, played the underlying clip inline, navigated to + the next neighboring frame (score panel updates: domain/visual/action/ + feedback) +- `/ask` — opened a fresh WebSocket chat session ("WebSocket connected"), + sent a new prompt about the just-uploaded video, and received a full + streamed RAG answer grounded in the retrieved frame captions + +## Notes + +- **Real Google SSO cannot be driven by the automation tooling** — the + browser tool refuses any interaction (including screenshots) on + `accounts.google.com`. The account-chooser/consent step was completed by + the user directly in the same Chrome tab; automation resumed once the + browser redirected back to `/library`. +- **Frame captions are motion-heuristic fallbacks** ("mostly static cctv + video segment", "low motion ... video segment"), not full VLM scene + descriptions — same behavior as the June 16 recording. The `/ask` chat + answer, by contrast, is a real Qwen2.5-VL generation grounded in retrieval + context, so the VLM path itself works; only the per-frame captioning step + is degraded (likely hitting the extractor's soft-timeout fallback under + local Ollama latency). +- Qdrant retained embeddings from a prior ingestion run (visible as the + `NTU_fight0160_gun_1.mp4` sourced hits) — the search results shown mix + frames from that older corpus with the newly uploaded video. +- **Build fix required to bring the stack up**: `MLService/inference/embedder/requirements.txt` + pinned `fastapi==0.118.0` with `starlette==0.49.1`, which is outside + fastapi 0.118's `starlette<0.49.0` ceiling — pip's resolver failed the + `embedder` image build. Bumped to `fastapi==0.121.0` (ceiling + `starlette<0.50.0`) to resolve. +- **Port 8080 conflict**: an unrelated `dataflow-control-plane` (kind/k8s) + container already held host port 8080. Stopped it (`docker stop + dataflow-control-plane`) to free the port for the `api` service, since + VideoVault's Google OAuth redirect URI is registered as + `http://localhost:8080/...` and can't be remapped without console access. + Restart it with `docker start dataflow-control-plane` when done. diff --git a/e2e-recordings/2026-07-25-user-workflow/manifest.json b/e2e-recordings/2026-07-25-user-workflow/manifest.json new file mode 100644 index 0000000..70ae96c --- /dev/null +++ b/e2e-recordings/2026-07-25-user-workflow/manifest.json @@ -0,0 +1,31 @@ +{ + "title": "VideoVault Real User Workflow Recording", + "createdAt": "2026-07-25T07:55:00Z", + "captureMode": "Claude in Chrome gif_creator (real browser session) -> GIF -> ffmpeg MP4", + "stack": "local docker compose (web, api, mongo, postgres, redis, qdrant, embedder, chunker, normalizer, extractor) + host Ollama qwen2.5vl:7b", + "uploadedVideo": "/Users/aryaman.sinha/Downloads/13258882-uhd_3840_2160_30fps.mp4", + "testedFlow": [ + "Signed-out landing page", + "Real Google OAuth2 SSO (manual, in-browser)", + "Authenticated /library (v2 UI)", + "Multipart upload of 86.4MB 4K video", + "Live SSE pipeline status: queued -> chunking -> normalized -> indexing -> Ready", + "Search at default 80% confidence: zero results, threshold-gating UX", + "Domain switch to 'CCTV adaptive demo'", + "Search 'low motion': 6 high-confidence frame hits", + "Frame modal: inline clip playback + next-frame navigation with live re-score", + "/ask: fresh WebSocket session, new prompt about uploaded video, full streamed RAG answer" + ], + "buildFixes": [ + { + "file": "MLService/inference/embedder/requirements.txt", + "issue": "fastapi==0.118.0 requires starlette<0.49.0 but starlette==0.49.1 was pinned, causing pip ResolutionImpossible", + "fix": "bumped fastapi to 0.121.0 (starlette<0.50.0 ceiling)" + } + ], + "environmentNotes": [ + "Stopped unrelated dataflow-control-plane container to free host port 8080 (VideoVault api's Google OAuth redirect_uri is hardcoded to localhost:8080)", + "Qdrant retained embeddings from a prior ingestion run; search results include hits from that older corpus alongside the newly uploaded video", + "Frame captions are motion-heuristic fallbacks, not full VLM scene descriptions; /ask chat answers are real Qwen2.5-VL generations" + ] +} diff --git a/e2e-recordings/2026-07-25-user-workflow/videovault-user-workflow-recording.gif b/e2e-recordings/2026-07-25-user-workflow/videovault-user-workflow-recording.gif new file mode 100644 index 0000000..077b164 Binary files /dev/null and b/e2e-recordings/2026-07-25-user-workflow/videovault-user-workflow-recording.gif differ diff --git a/e2e-recordings/2026-07-25-user-workflow/videovault-user-workflow-recording.mp4 b/e2e-recordings/2026-07-25-user-workflow/videovault-user-workflow-recording.mp4 new file mode 100644 index 0000000..9358c96 Binary files /dev/null and b/e2e-recordings/2026-07-25-user-workflow/videovault-user-workflow-recording.mp4 differ diff --git a/e2e-recordings/2026-07-29-user-persona-demo/README.md b/e2e-recordings/2026-07-29-user-persona-demo/README.md new file mode 100644 index 0000000..25db825 --- /dev/null +++ b/e2e-recordings/2026-07-29-user-persona-demo/README.md @@ -0,0 +1,98 @@ +# VideoVault User Persona / Few-Shot Domain Demo — 2026-07-29 + +Created: 2026-07-29T18:20:00Z + +Capture mode: Claude in Chrome `gif_creator` (real Chrome, real Google +session) → GIF (50 frames, click indicators + action labels) → re-encoded to +MP4 with ffmpeg. + +**Narrated version**: +[`videovault-user-persona-demo-recording-narrated.mp4`](videovault-user-persona-demo-recording-narrated.mp4) +(78.8s). Built as 6 independently-captured, independently-narrated segments +rather than one voiceover laid over the raw recording — the first attempt at +that (voiceover stretched to cover a single continuous capture) put the audio +out of sync with what was on screen, and the raw capture's frame timing isn't +proportional to real elapsed time in the first place. Instead, each of the 6 +milestones below was re-captured live as its own single-frame screenshot +(app state is fully server-persisted and reproducible), narrated with its own +`say -v Samantha` clip +([`narration-script.txt`](narration-script.txt), one paragraph per segment), +and each segment's still image is held on screen for exactly its narration's +duration (`ffmpeg -loop 1 -t `) before cutting to the +next — guaranteeing narration and visual are aligned at every segment +boundary, and guaranteeing the final segment shows the actual final state +(not a frozen frame that happened to be the last one captured, which was +wrong in the first attempt — it showed a stale/earlier domain). + +Real testing dataset used throughout: the +[Kaggle CCTV Action Recognition dataset](https://www.kaggle.com/datasets/jonathannield/cctv-action-recognition-dataset) +already ingested into this account's library in a prior benchmark session — +2,300 real CCTV clips across 13 action classes (`fall`, `grab`, `gun`, `hit`, +`kick`, `lying_down`, `run`, `sit`, `sneak`, `stand`, `struggle`, `throw`, +`walk`). See `tests/bench/data/Videos/Videos/` and `tests/bench/queries.yaml`. + +## What this demonstrates + +**1. Normal (baseline) search on a specific instance query.** `Domain = +General`, query `"a person holding a gun"` (a specific action, not a generic +term like "low motion") → **zero results**, even after lowering the +confidence threshold. Generic semantic search over this corpus's +motion-heuristic fallback captions cannot find a specific action. + +**2. Few-shot domain / user persona — how it's built.** VideoVault's +"Domains" feature (`/features`, "Manage learning" from the library) is the +persona mechanism: +- Created a fresh domain, `CCTV Crime Analyst` (name + context), to show the + onboarding flow from zero: *Create a domain → Add search labels → Upload + 3-5 examples per label → Run searches and rate results*. +- Added a label `gun` with model context `"a person holding or brandishing a + gun"` via the Few-shot labels panel. Domain state after this: `1 labels, 0 + examples` — a domain with a label but no training signal. +- Confirmed a brand-new domain with 0 examples performs **identically to + General** (also zero results for the same query) — creating a domain by + itself does nothing; it needs real few-shot signal. + +**3. The already-trained persona.** A domain from a prior session, +`CCTV adaptive demo`, already has real few-shot training: **3 labels, 23 +examples** (`fall · 12`, `gun · 9`, `shooting · 1`), built from actual +uploaded example clips + search-feedback (thumbs up/down), which the product +describes as: *"Search feedback updates positive and negative centroids for +this domain; examples provide the few-shot prototype signal."* + +**4. Persona search vs. normal search, same specific query.** `Domain = CCTV +adaptive demo`, same query `"a person holding a gun"` → **24/24 results at +≥80% confidence**, every hit tagged `Action gun`, top hit +`NTU_fight0222_gun_1.mp4` at 86% confidence (`Visual 100%`, `Action 95%`) — a +real CCTV clip of armed men. Zero → 24 for the identical query, General vs. +the trained persona domain. + +**5. Adding a label live / making search more efficient.** Thumbs-upped a +correct `gun` hit from the result list while `CCTV adaptive demo` was active. +UI confirmed: *"Saved. Future results adapt to this domain."* Domain's +`Feedback` counter moved `2 → 3` and the model version bumped +(`v1782239 → v1785349`) — the live active-learning loop, in real time. + +## File upload constraint (why the new domain wasn't also trained with fresh uploads) + +The `Upload examples` few-shot flow requires a native file picker; automated +`file_upload` is restricted to files the user explicitly shared with this +session (chat attachments), not arbitrary paths on disk — including files +already inside the project's own test dataset. Raw file-based few-shot +training therefore has to be done by hand through the UI or via the +`tests/bench/run_domain_adaptation_experiments.py` benchmark script (which +authenticates via `BENCHMARK_AUTH_ENABLED` + `X-Benchmark-Google-Sub`, a +*separate* synthetic account — its uploads don't appear in a real user's +library). The **search-feedback path** (thumbs up/down on real results) has +no such restriction and was used instead to demonstrate live label +strengthening. + +## Notes + +- Frame captions across this corpus are still motion-heuristic fallbacks + ("mostly static", "low motion ... segment"), not full VLM scene + descriptions — same caveat as the `2026-07-25-user-workflow` recording. + The persona domain's `Action gun` / `Visual` / `Domain` score components + (not the caption text) are what actually carry the gun-specific signal. +- Google SSO was completed manually by the user in-browser twice (session + expired mid-recording) — real automation cannot drive `accounts.google.com` + by design. diff --git a/e2e-recordings/2026-07-29-user-persona-demo/manifest.json b/e2e-recordings/2026-07-29-user-persona-demo/manifest.json new file mode 100644 index 0000000..0827f26 --- /dev/null +++ b/e2e-recordings/2026-07-29-user-persona-demo/manifest.json @@ -0,0 +1,28 @@ +{ + "title": "VideoVault User Persona / Few-Shot Domain Demo", + "createdAt": "2026-07-29T18:20:00Z", + "captureMode": "Claude in Chrome gif_creator (real browser session) -> GIF -> ffmpeg MP4", + "dataset": "Kaggle CCTV Action Recognition dataset (tests/bench/data/Videos/Videos), 2300 clips, 13 action classes", + "testedFlow": [ + "Baseline: Domain=General, query 'a person holding a gun' -> 0 results even at lowered confidence", + "Created new domain 'CCTV Crime Analyst' via /features onboarding flow", + "Added few-shot label 'gun' with model context -> domain state: 1 labels, 0 examples", + "Confirmed untrained new domain performs identically to General (0 results) for same query", + "Switched to pre-trained domain 'CCTV adaptive demo' (3 labels, 23 examples: fall.12 gun.9 shooting.1)", + "Same query 'a person holding a gun' under trained persona -> 24/24 results >=80% confidence, all tagged Action gun", + "Top hit NTU_fight0222_gun_1.mp4 at 86% confidence (Visual 100%, Action 95%)", + "Live feedback: thumbs-up on a correct hit -> 'Saved. Future results adapt to this domain.', Feedback count 2->3, model version bumped" + ], + "comparison": { + "query": "a person holding a gun", + "general_domain_results": 0, + "untrained_new_domain_results": 0, + "trained_persona_domain_results": 24, + "trained_persona_top_confidence": "86%" + }, + "constraints": [ + "file_upload tool restricted to session-shared files only; could not upload fresh few-shot example clips from the local test dataset even though they're in the project directory", + "Used search-feedback (thumbs up/down) as the live few-shot signal instead, which has no such restriction", + "Real Google SSO completed manually by user in-browser (2x, session expired once mid-recording); accounts.google.com cannot be automated" + ] +} diff --git a/e2e-recordings/2026-07-29-user-persona-demo/narration-script.txt b/e2e-recordings/2026-07-29-user-persona-demo/narration-script.txt new file mode 100644 index 0000000..4d22a55 --- /dev/null +++ b/e2e-recordings/2026-07-29-user-persona-demo/narration-script.txt @@ -0,0 +1,6 @@ +This is VideoVault. We start with a normal search under the General domain, asking for a very specific action: a person holding a gun. Zero results, even at the default threshold. Generic search struggles with a specific instance like this. +So we build a user persona. On the Features page, we create a new domain, CCTV Crime Analyst, and add a few-shot label for gun, with a short description. +But a brand new domain with zero examples behaves exactly like General: still zero results. Creating a domain alone isn't enough. It needs real training signal. +So we switch to an already-trained persona, CCTV adaptive demo, built earlier from real few-shot examples, including nine for gun. Same exact query: twenty four out of twenty four results, all tagged Action gun. +Here's the top hit, at eighty eight percent confidence. Domain match and Visual match both at one hundred percent, Action at ninety five percent. A real CCTV clip of armed men. +Finally, the live feedback loop. A thumbs-up on a correct result saves instantly, adapting the domain in real time. Feedback count and model version update immediately. That's few-shot domain adaptation: zero results with generic search, twenty four relevant hits with a trained persona, and search that keeps getting better the more you label. diff --git a/e2e-recordings/2026-07-29-user-persona-demo/videovault-user-persona-demo-recording-narrated.mp4 b/e2e-recordings/2026-07-29-user-persona-demo/videovault-user-persona-demo-recording-narrated.mp4 new file mode 100644 index 0000000..f4ac2a6 Binary files /dev/null and b/e2e-recordings/2026-07-29-user-persona-demo/videovault-user-persona-demo-recording-narrated.mp4 differ diff --git a/e2e-recordings/2026-07-29-user-persona-demo/videovault-user-persona-demo-recording.gif b/e2e-recordings/2026-07-29-user-persona-demo/videovault-user-persona-demo-recording.gif new file mode 100644 index 0000000..4b9d612 Binary files /dev/null and b/e2e-recordings/2026-07-29-user-persona-demo/videovault-user-persona-demo-recording.gif differ diff --git a/e2e-recordings/2026-07-29-user-persona-demo/videovault-user-persona-demo-recording.mp4 b/e2e-recordings/2026-07-29-user-persona-demo/videovault-user-persona-demo-recording.mp4 new file mode 100644 index 0000000..100c8ee Binary files /dev/null and b/e2e-recordings/2026-07-29-user-persona-demo/videovault-user-persona-demo-recording.mp4 differ