From e5a5198a6fc1409a0f502ccabe869c6af7199abd Mon Sep 17 00:00:00 2001 From: "Brian.oh" <49855381+dev07060@users.noreply.github.com> Date: Mon, 1 Jun 2026 02:06:03 +0900 Subject: [PATCH 1/2] docs(perf): on-device query profiler design spec (LOC-65) Brainstorming output for a real-device RAG query-latency profiler. Approach C (phased): coarse Dart-only baseline (no Rust change) then drill into the dominant bucket. Grounded in verified architecture facts: one-active-collection singleton, collection-switch rebuilds BM25 (full re-tokenize, no disk persist) + reloads HNSW from disk (Box::leak), active collection has zero per-query BM25 tokenization (0.18.4), filtered search routes to the i8 exact-scan kernel. Adds docs/perf/ondevice-query-profiler/{DESIGN.md,README.md}. Mirrors the vector-math-refactor journal pattern; Linear project + LOC-65 created. --- docs/perf/ondevice-query-profiler/DESIGN.md | 71 +++++++++++++++++++++ docs/perf/ondevice-query-profiler/README.md | 26 ++++++++ 2 files changed, 97 insertions(+) create mode 100644 docs/perf/ondevice-query-profiler/DESIGN.md create mode 100644 docs/perf/ondevice-query-profiler/README.md diff --git a/docs/perf/ondevice-query-profiler/DESIGN.md b/docs/perf/ondevice-query-profiler/DESIGN.md new file mode 100644 index 0000000..7064b14 --- /dev/null +++ b/docs/perf/ondevice-query-profiler/DESIGN.md @@ -0,0 +1,71 @@ +# On-Device RAG Query Profiler — Design Spec + +- 작성: 2026-06-01 +- 상태: 설계 승인됨 (스펙 리뷰 대기) +- Linear: [LOC-65](https://linear.app/loceract/issue/LOC-65) · 프로젝트 [온디바이스 RAG 쿼리 프로파일러](https://linear.app/loceract/project/온디바이스-rag-쿼리-프로파일러-25df240c4262) +- 1차 목표: **실기에서 쿼리 1회의 단계별 지연을 분해**해 진짜 병목을 찾고, baseline을 박제(재실행/비교 가능)한다. + +## 1. 배경 / 검증된 아키텍처 사실 +직전 결론: vector_math 커널 슬라이스는 거의 최적이나, **온디바이스 RAG end-to-end는 미측정**이며 병목은 십중팔구 임베딩 추론 등 다른 곳. 이 프로파일러로 그걸 데이터로 확정한다. + +설계의 전제는 모두 코드로 검증함: +- **One-Active-Collection 싱글톤**: `ACTIVE_HNSW_COLLECTION`/`ACTIVE_BM25_COLLECTION: RwLock>` ([source_rag.rs:55-56](../../../rust_builder/rust/src/api/source_rag.rs#L55)) + 단일 전역 `INVERTED_INDEX`([bm25_search.rs:24](../../../rust_builder/rust/src/api/bm25_search.rs#L24)) · 단일 전역 `HNSW_INDEX`([hnsw_index.rs:47](../../../rust_builder/rust/src/api/hnsw_index.rs#L47)). 한 번에 한 컬렉션만 warm. +- **컬렉션 스위치 비용은 별도 명시 호출 `activateCollectionForHybridSearch`에 있음** ([source_rag_service.dart:948/1453](../../../lib/services/source_rag_service.dart#L948)), `search_meta_hybrid` 본문에 숨지 않음. 활성화 시: BM25를 `bm25_clear_index()` 후 컬렉션 전 청크 재토큰화로 **디스크 영속 없이 재구축**([source_rag.rs:930-965](../../../rust_builder/rust/src/api/source_rag.rs#L930)); HNSW는 디스크 `load_hnsw_index`(또는 없으면 rebuild) + `Box::leak`([hnsw_index.rs:196](../../../rust_builder/rust/src/api/hnsw_index.rs#L196), [source_rag.rs:1018-1026](../../../rust_builder/rust/src/api/source_rag.rs#L1018)). +- **활성 컬렉션 내 쿼리는 BM25 재토큰화 0회** (0.18.4 픽스, term stats가 사전구축 active index에서 옴 — [hybrid_search.rs:215-218](../../../rust_builder/rust/src/api/hybrid_search.rs#L215)). +- **Filtered(source_ids) 검색은 i8 exact-scan 핫커널로 라우팅** ([hybrid_search.rs:183](../../../rust_builder/rust/src/api/hybrid_search.rs#L183) "switching to exact scan"). +- 반복 스위치 시 `Box::leak`로 메모리 증가 — 메모리 측정 시 증가 지점(현 범위 밖, 기록만). + +## 2. 목표 / 비목표 +**In**: 쿼리 지연 단계분해(Phase1 coarse) + 조건부 Phase2 드릴다운, 컬렉션 스위치(A→B→A) 지터 측정, JSON/CSV export + 구조화 로그, 연결된 1대, 출시 설정(`vector_faer,vector_quant_i8`). +**Out(후속)**: 인덱싱 플로우 프로파일, 메모리/배터리, CI 회귀 게이트, iOS↔Android 비교, recall/품질 검증. + +## 3. 접근법 — Approach C (단계적) +Phase1(coarse, Rust 0변경)로 큰 그림 + baseline → 데이터가 가리키는 버킷만 Phase2로 드릴다운. "측정 먼저, 입증된 곳만 손댄다" 원칙의 재귀 적용. + +## 4. 측정 세그먼트 (쿼리/활성화 경로) +- `embed`: query 임베딩(ONNX, Dart) — Stopwatch +- `activate`: `activateCollectionForHybridSearch` — Stopwatch. **cold/switch에서 bm25-rebuild vs hnsw-load 의무 분해**(Phase2-activation) +- `search`: search FFI 총시간 — Stopwatch (warm steady-state에서 지배 시 Phase2-search로 ANN vs BM25-rank vs RRF 분해) +- `hydrate`: 콘텐츠 hydrate — Stopwatch +- I/O 카운터: `query_metrics` 스냅샷(쿼리 전후) — rows/bytes/tokenization nanos ([query_metrics.rs](../../../rust_builder/rust/src/api/query_metrics.rs)) +- **FFI 오버헤드 수식(저널 명시)**: `Dart Stopwatch(segment) − Rust 내부 계측(segment) = 순수 FFI 마샬링 + Isolate 컨텍스트 스위치 비용` + +## 5. 쿼리 2레인 +- **Lane 1 Unfiltered**: 프로덕션 기본 패스 = f32 HNSW + BM25 via RRF. +- **Lane 2 Filtered**: `source_ids` 강제 주입 → i8 exact-scan 핫커널 (DRAM-bound 여부 실측). + +## 6. Cold/Warm 분류 + 시나리오 +- **Pure Cold**: 앱 최초 실행 후 첫 쿼리 (ONNX init + SQLite page cache cold). +- **Switching Cold**: 컬렉션 B 활성 상태에서 A로 전환 직후 첫 쿼리 — `activate` 세그먼트가 BM25 전체 재토큰화 + HNSW 디스크 재로드를 흡수. +- **Pure Warm**: 동일 컬렉션 연속 쿼리(2번째~). +- 필수 시나리오: **A 활성 → B 활성 → A 쿼리**(inter-collection 지터). 순수 warm만 재면 "가짜 초록"이 되므로 스위치 시나리오는 In. + +## 7. Phase 게이트 (두 종류 분리) +- **Phase2-activation (cold/switch 의무)**: `activate`를 bm25-rebuild vs hnsw-load로 분해. Rust 계측 또는 기존 로그 타임스탬프 활용. +- **Phase2-search (조건부)**: warm steady-state에서 `search`가 지배하면 Rust `QueryTimings`(thread-local Instant, `query_metrics` 패턴, FFI 스냅샷)로 ANN/BM25-rank/RRF 분해. +- **embed 지배 시**: Rust 계측 불필요 → 다음 대상은 추론(ONNX, 크레이트 밖). 그 사실이 결론. + +## 8. 컴포넌트 +- **고정 픽스처**: 결정적 코퍼스(시드 N청크) + 최소 2개 컬렉션(A/B, 스위치용) + 고정 쿼리셋(M개, 두 레인). 출시 feature. +- **프로파일 드라이버**(`integration_test/query_profile_test.dart`): warmup→measured 루프, 세그먼트 측정 + `query_metrics` 스냅샷. `benchmark_service.dart`의 `collectSamples`/`_percentileFromSorted`/`_stdDev` 재사용. +- **결과 모델 + 익스포터**: `QueryProfileSample`(레인/카테고리/세그먼트별 ms, I/O 카운터, 실행 메타) → app docs dir에 JSON+CSV, 실행당 로그 1줄. 가능하면 `benchmark_models.dart` 재사용. +- **(Phase2-search 조건부) Rust `QueryTimings`**: search 내부 단계 Instant + FFI 스냅샷/reset. + +## 9. 데이터 흐름 (측정 쿼리 1회) +`query_metrics reset → [필요시 activate(A) 측정] → t0 → embed(query) → t1 → search(emb,top_k,filter?) → t2 → hydrate(top_k) → t3 → metrics snapshot`. 세그먼트 = embed/activate/search/hydrate. N회 후 카테고리·레인별 p50/p95/mean/stddev 집계 → export. + +## 10. 방법론 / 타당성 +- warmup 폐기(JIT·파일캐시·ONNX 세션·page cache). Pure Cold/Switching Cold는 별도 1회성 측정으로 분리 기록. +- 기기 상태 export 메타에 기록: **충전 연결(써멀 스로틀 방지)**, 화면 ON, 백그라운드 최소화. +- fail-closed: measured N>0 & 각 세그먼트 기록 검증(PDF 스모크식). 원시 샘플 JSON 보존. +- 오버헤드: Stopwatch/atomics는 ms 스케일 대비 무시 가능 — 명시. +- export 메타: 기기/OS, 코퍼스 크기, top_k, feature 플래그, 레인, 카테고리, HNSW/exact, 충전상태. + +## 11. 산출물 (export 스키마 핵심 필드) +`{device, os, ts, corpus_size, top_k, features, lane, category, segments_ms:{embed,activate,hydrate,search}, search_breakdown_ms?:{ann,bm25,rrf}, activate_breakdown_ms?:{bm25_rebuild,hnsw_load}, io:{rows,bytes,tok_nanos}, ffi_overhead_ms, samples:[...], p50/p95/mean/stddev per segment}` + +## 12. 테스트 (하니스 자체 검증) +세그먼트 합 ≈ end-to-end 총시간(sanity), 결정적 코퍼스→재현 카운트, 연결 기기에서 green, fail-closed 동작. + +## 13. 미해결/후속 (Out) +인덱싱 플로우, 메모리(Box::leak 누수 포함)·배터리, CI 회귀 게이트, iOS↔Android 비교, recall/품질. 1차(프로파일링) 종료 후 데이터 기반 재평가. diff --git a/docs/perf/ondevice-query-profiler/README.md b/docs/perf/ondevice-query-profiler/README.md new file mode 100644 index 0000000..87a5ab4 --- /dev/null +++ b/docs/perf/ondevice-query-profiler/README.md @@ -0,0 +1,26 @@ +# 온디바이스 RAG 쿼리 프로파일러 — 작업 저널 + +> 직전 vector-math-refactor와 동일 방식: PR(히스토리) 단위로 결과·피드백 보존, Linear 미러링. + +## 배경 한 줄 +vector_math 커널 슬라이스는 거의 최적임을 확인했으나 **온디바이스 RAG end-to-end는 미측정**. 실기에서 쿼리 단계별 지연을 분해해 진짜 병목을 데이터로 찾는다. + +## 문서 +- [DESIGN.md](DESIGN.md) — 승인된 설계 스펙(검증된 아키텍처 사실 포함). 구현 계획은 writing-plans 산출 후 추가. +- Linear 프로젝트: [온디바이스 RAG 쿼리 프로파일러](https://linear.app/loceract/project/온디바이스-rag-쿼리-프로파일러-25df240c4262) · 설계 이슈 [LOC-65](https://linear.app/loceract/issue/LOC-65) + +## 핵심 설계 결론 (요약) +- Approach C(단계적): Phase1 coarse(Rust 0변경) baseline → 지배 버킷만 Phase2 드릴다운. +- 측정 세그먼트: embed / **activate(스위치 비용)** / search / hydrate + I/O 카운터 + FFI 오버헤드 수식. +- 2레인: Unfiltered(f32 HNSW+BM25 RRF) / Filtered(i8 exact-scan). +- Cold/Warm 3분류 + **A→B→A 스위치 시나리오 필수**(순수 warm만 재면 가짜 초록). + +## PR 분할 / 상태 +(구현 계획 확정 후 채움 — 직전 저널 형식과 동일한 상태표·PRn.md) + +| PR | 제목 | Linear | 상태 | +|----|------|--------|------| +| — | 설계 스펙 (이 문서) | [LOC-65](https://linear.app/loceract/issue/LOC-65) | 🟦 진행 | + +## 규약 (프로젝트 공통) +- CI: `cargo test -- --test-threads=1`. 커밋/PR에 Claude 귀속 미포함. PR은 열고 CI green까지만, 머지는 본인. From 41034ccec1e2c584ef4682a3c84ea2e0944e03f2 Mon Sep 17 00:00:00 2001 From: "Brian.oh" <49855381+dev07060@users.noreply.github.com> Date: Mon, 1 Jun 2026 02:15:24 +0900 Subject: [PATCH 2/2] docs(perf): add query profiler implementation plan (LOC-65) PR-sized decomposition (P1-P5, LOC-66..70) with TDD steps and verified Dart/Rust signatures. Phase 1 fully specified (host-TDD utils + device integration_test in the example app, 2 lanes, 3 cold/warm scenarios incl. A->B->A switch, JSON/CSV export); Phase 2 gated on baseline data. Mirrors vector-math-refactor journal; Linear LOC-66..70 created. --- docs/perf/ondevice-query-profiler/PLAN.md | 615 ++++++++++++++++++++ docs/perf/ondevice-query-profiler/README.md | 10 +- 2 files changed, 623 insertions(+), 2 deletions(-) create mode 100644 docs/perf/ondevice-query-profiler/PLAN.md diff --git a/docs/perf/ondevice-query-profiler/PLAN.md b/docs/perf/ondevice-query-profiler/PLAN.md new file mode 100644 index 0000000..7dbe2e2 --- /dev/null +++ b/docs/perf/ondevice-query-profiler/PLAN.md @@ -0,0 +1,615 @@ +# On-Device RAG Query Profiler — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Build a repeatable on-device Flutter `integration_test` that decomposes RAG query latency into stages (embed / activate / search / hydrate), captures I/O counters + an FFI-overhead figure, and exports p50/p95 to JSON+CSV so the real bottleneck is found from data. + +**Architecture:** Approach C (phased). Phase 1 = coarse, Dart-only segments (zero Rust change) reusing `benchmark_service.dart` stats utilities; runs in the **example app** (which bundles the real ONNX model + tokenizer assets) so `embed` is real. Phase 2 = conditional drill-down into whichever bucket dominates. Spec: [DESIGN.md](DESIGN.md). + +**Tech Stack:** Flutter `integration_test`, Dart `Stopwatch`, existing `EmbeddingService` / `SourceRagService` / `query_metrics` FFI, `dart:convert` + `dart:io` for export. + +**Verified-fact anchors (do not re-derive):** one-active-collection singleton; `activateCollectionForHybridSearch` is a *separate* call (switch cost lives there, NOT inside `searchMetaHybrid`); active-collection queries do zero BM25 re-tokenization (0.18.4); filtered search → i8 exact-scan. See DESIGN §1. + +--- + +## PR Decomposition (mirrors vector-math-refactor journal; each → a Linear issue) + +| PR | Title | Linear | Host-testable? | +|----|-------|--------|----------------| +| P1 | Stats + report model + export utils (pure, host-TDD) | LOC-66 | ✅ yes (`flutter test`) | +| P2 | example integration_test wiring + A/B fixture builder | LOC-67 | device | +| P3 | Segment timing loop + 3 scenarios + query_metrics snapshot | LOC-68 | device | +| P4 | JSON/CSV export to app-docs + structured log + run metadata | LOC-69 | device + host(serialize) | +| P5 | (conditional) Phase-2 drill-down per dominant bucket | LOC-70 | TBD by P3/P4 data | + +Each PR: branch from main, commit (no Claude attribution), open PR, CI green (`cargo test -- --test-threads=1` unaffected — Dart-only), **user merges**. Fill `PRn.md` + README row before merge. + +--- + +## File Structure + +- Create `lib/services/benchmark_service.dart` → **add** one public method `statsFromSamples` (reuses existing private `_percentileFromSorted`/`_stdDev`). (P1) +- Create `example/lib/profiling/query_profile_report.dart` — pure model + JSON/CSV serializers + FFI-overhead computation. (P1) +- Create `example/lib/profiling/query_fixture.dart` — deterministic A/B corpus + 2-lane query set. (P2) +- Create `example/lib/profiling/query_profiler.dart` — the segment-timing driver (embed/activate/search/hydrate + metrics snapshot). (P3) +- Create `example/integration_test/query_profile_test.dart` — entrypoint: init engine, build fixture, run scenarios, export. (P2 skeleton → P3/P4 fill) +- Modify `example/pubspec.yaml` — add `integration_test` dev-dep. (P2) +- Create `test/unit/query_profile_report_test.dart` + `test/unit/benchmark_stats_test.dart` — host unit tests. (P1) + +--- + +## Task P1.1: Public stats-from-samples helper + +**Files:** +- Modify: `lib/services/benchmark_service.dart` (add method; reuse existing private helpers at lines 29-54) +- Test: `test/unit/benchmark_stats_test.dart` + +- [ ] **Step 1: Write failing test** +```dart +// test/unit/benchmark_stats_test.dart +import 'package:flutter_test/flutter_test.dart'; +import 'package:mobile_rag_engine/services/benchmark_service.dart'; +import 'package:mobile_rag_engine/models/benchmark_models.dart'; + +void main() { + test('statsFromSamples computes p50/p95/avg/stddev', () { + final s = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]; + final DetailedBenchmarkStats st = + BenchmarkService.statsFromSamples(s, warmupIterations: 0, measuredIterations: s.length); + expect(st.measuredIterations, 10); + expect(st.minMs, 1); + expect(st.maxMs, 10); + expect(st.avgMs, closeTo(5.5, 1e-9)); + expect(st.p50Ms, closeTo(5.5, 1e-9)); // (10-1)*0.5 = 4.5 → interp between 5 and 6 + expect(st.p95Ms, closeTo(9.55, 1e-9)); // (10-1)*0.95 = 8.55 → between 9 and 10 + expect(st.stdDevMs, greaterThan(0)); + }); + + test('statsFromSamples handles empty', () { + final st = BenchmarkService.statsFromSamples([], warmupIterations: 0, measuredIterations: 0); + expect(st.p50Ms, 0.0); + expect(st.p95Ms, 0.0); + }); +} +``` + +- [ ] **Step 2: Run test, verify it fails** +Run: `flutter test test/unit/benchmark_stats_test.dart` +Expected: FAIL — `statsFromSamples` not defined. + +- [ ] **Step 3: Implement (add to `class BenchmarkService`)** +```dart +/// Build DetailedBenchmarkStats from already-collected samples (ms). +/// Reuses the existing percentile/stddev helpers. +static DetailedBenchmarkStats statsFromSamples( + List samples, { + required int warmupIterations, + required int measuredIterations, +}) { + if (samples.isEmpty) { + return DetailedBenchmarkStats( + warmupIterations: warmupIterations, + measuredIterations: measuredIterations, + samplesMs: const [], + avgMs: 0, minMs: 0, maxMs: 0, p50Ms: 0, p95Ms: 0, stdDevMs: 0, + ); + } + final sorted = [...samples]..sort(); + final avg = samples.reduce((a, b) => a + b) / samples.length; + return DetailedBenchmarkStats( + warmupIterations: warmupIterations, + measuredIterations: measuredIterations, + samplesMs: samples, + avgMs: avg, + minMs: sorted.first, + maxMs: sorted.last, + p50Ms: _percentileFromSorted(sorted, 0.50), + p95Ms: _percentileFromSorted(sorted, 0.95), + stdDevMs: _stdDev(samples, avg), + ); +} +``` + +- [ ] **Step 4: Run test, verify PASS** +Run: `flutter test test/unit/benchmark_stats_test.dart` → Expected: PASS. + +- [ ] **Step 5: Commit** +```bash +git add lib/services/benchmark_service.dart test/unit/benchmark_stats_test.dart +git commit -m "feat(profiling): public statsFromSamples on BenchmarkService (LOC-66)" +``` + +--- + +## Task P1.2: Query profile report model (segments + FFI-overhead + JSON/CSV) + +**Files:** +- Create: `example/lib/profiling/query_profile_report.dart` +- Test: `test/unit/query_profile_report_test.dart` + +- [ ] **Step 1: Write failing test** +```dart +// test/unit/query_profile_report_test.dart +import 'package:flutter_test/flutter_test.dart'; +import 'package:rag_engine_example/profiling/query_profile_report.dart'; + +void main() { + test('SegmentSamples aggregates and serializes', () { + final seg = SegmentSamples('embed')..addAll([10, 12, 11, 13, 9]); + final j = seg.toJson(); + expect(j['segment'], 'embed'); + expect(j['p50_ms'], isNotNull); + expect((j['samples_ms'] as List).length, 5); + }); + + test('ffiOverheadMs = dart segment minus rust internal', () { + expect(QueryProfileRun.ffiOverheadMs(dartMs: 5.0, rustInternalMs: 3.2), closeTo(1.8, 1e-9)); + // never negative (clock noise) -> clamp to 0 + expect(QueryProfileRun.ffiOverheadMs(dartMs: 3.0, rustInternalMs: 3.4), 0.0); + }); + + test('report toCsv has one row per (lane,category,segment)', () { + final run = QueryProfileRun( + lane: 'unfiltered', category: 'pure_warm', + segments: { + 'embed': SegmentSamples('embed')..addAll([10, 11]), + 'search': SegmentSamples('search')..addAll([2, 3]), + }, + io: const {'scoped_exact_scan_rows': 0}, + meta: const {'device': 'test'}, + ); + final report = QueryProfileReport(runs: [run]); + final csv = report.toCsv(); + expect(csv, contains('lane,category,segment,p50_ms,p95_ms,avg_ms,min_ms,max_ms,stddev_ms,n')); + expect(csv, contains('unfiltered,pure_warm,embed,')); + expect(csv, contains('unfiltered,pure_warm,search,')); + final map = report.toJson(); + expect((map['runs'] as List).length, 1); + }); +} +``` + +- [ ] **Step 2: Run, verify FAIL** +Run: `flutter test test/unit/query_profile_report_test.dart` → FAIL (library missing). +> Note: `rag_engine_example` is the example app package name — confirm in `example/pubspec.yaml` `name:` and use it in the import. + +- [ ] **Step 3: Implement** +```dart +// example/lib/profiling/query_profile_report.dart +import 'dart:convert'; +import 'package:mobile_rag_engine/services/benchmark_service.dart'; +import 'package:mobile_rag_engine/models/benchmark_models.dart'; + +/// Per-segment sample collector that defers stats to BenchmarkService. +class SegmentSamples { + final String segment; + final List samplesMs = []; + SegmentSamples(this.segment); + void add(double ms) => samplesMs.add(ms); + void addAll(Iterable ms) => samplesMs.addAll(ms); + + DetailedBenchmarkStats stats() => BenchmarkService.statsFromSamples( + samplesMs, warmupIterations: 0, measuredIterations: samplesMs.length); + + Map toJson() { + final s = stats(); + return {'segment': segment, ...s.toJson()}; + } +} + +/// One scenario run: a (lane, category) cell with its measured segments. +class QueryProfileRun { + final String lane; // 'unfiltered' | 'filtered' + final String category; // 'pure_cold' | 'switching_cold' | 'pure_warm' + final Map segments; + final Map io; // query_metrics snapshot (rows/bytes/nanos) + final Map meta; // device/config + + QueryProfileRun({ + required this.lane, required this.category, + required this.segments, required this.io, required this.meta, + }); + + /// FFI marshalling + isolate context-switch cost = Dart-measured − Rust-internal. + /// Clamped to 0 (clock noise can make it slightly negative). + static double ffiOverheadMs({required double dartMs, required double rustInternalMs}) { + final d = dartMs - rustInternalMs; + return d > 0 ? d : 0.0; + } + + Map toJson() => { + 'lane': lane, + 'category': category, + 'segments': segments.map((k, v) => MapEntry(k, v.toJson())), + 'io': io, + 'meta': meta, + }; +} + +class QueryProfileReport { + final List runs; + QueryProfileReport({required this.runs}); + + Map toJson() => {'runs': runs.map((r) => r.toJson()).toList()}; + String toJsonString() => const JsonEncoder.withIndent(' ').convert(toJson()); + + String toCsv() { + final b = StringBuffer() + ..writeln('lane,category,segment,p50_ms,p95_ms,avg_ms,min_ms,max_ms,stddev_ms,n'); + for (final r in runs) { + for (final seg in r.segments.values) { + final s = seg.stats(); + b.writeln('${r.lane},${r.category},${seg.segment},' + '${s.p50Ms},${s.p95Ms},${s.avgMs},${s.minMs},${s.maxMs},${s.stdDevMs},' + '${seg.samplesMs.length}'); + } + } + return b.toString(); + } +} +``` + +- [ ] **Step 4: Run, verify PASS** +Run: `flutter test test/unit/query_profile_report_test.dart` → PASS. + +- [ ] **Step 5: Commit** +```bash +git add example/lib/profiling/query_profile_report.dart test/unit/query_profile_report_test.dart +git commit -m "feat(profiling): query profile report model + JSON/CSV (LOC-66)" +``` + +> **PR P1 ends here.** Open PR, fill `PR-P1.md` + README row, user merges. + +--- + +## Task P2.1: Add integration_test dependency to the example app + +**Files:** Modify `example/pubspec.yaml` + +- [ ] **Step 1:** Add under `dev_dependencies:` +```yaml + integration_test: + sdk: flutter +``` +- [ ] **Step 2:** Run `cd example && flutter pub get` → Expected: resolves, no errors. +- [ ] **Step 3: Commit** +```bash +git add example/pubspec.yaml example/pubspec.lock +git commit -m "build(example): add integration_test dev-dep (LOC-67)" +``` + +## Task P2.2: Deterministic A/B fixture + 2-lane query set + +**Files:** Create `example/lib/profiling/query_fixture.dart` + +- [ ] **Step 1: Implement** (deterministic text; ingest builds chunks+embeddings+indexes) +```dart +// example/lib/profiling/query_fixture.dart +import 'package:mobile_rag_engine/mobile_rag_engine.dart'; + +class QueryFixture { + static const collectionA = 'profile_a'; + static const collectionB = 'profile_b'; + + /// Deterministic short docs. `count` controls corpus size for the run. + static List docs(String seedTag, int count) => List.generate( + count, + (i) => 'doc $seedTag $i: mobile retrieval augmented generation ' + 'embedding vector search bm25 ranking topic${i % 17} ' + 'token${(i * 7) % 53} alpha beta gamma delta epsilon', + ); + + /// Two query lanes. Lane 2 (filtered) injects sourceIds at call time. + static const unfilteredQueries = [ + 'vector search ranking', 'embedding topic3 retrieval', + 'bm25 token alpha', 'mobile generation gamma', 'topic9 delta epsilon', + ]; + + /// Seed both collections. Returns the source ids per collection (for filtered lane). + static Future>> seed({required int docsPerCollection}) async { + final ids = >{}; + for (final c in [collectionA, collectionB]) { + final col = MobileRag.instance.inCollection(c); + final srcIds = []; + for (final d in docs(c, docsPerCollection)) { + final r = await col.addDocumentUtf8(content: d); // confirm exact arg name in source_rag_service.inCollection().addDocumentUtf8 + srcIds.add(r.sourceId); // confirm field on the add-result type + } + ids[c] = srcIds; + } + return ids; + } +} +``` +> Execution note: confirm `addDocumentUtf8` parameter name + the returned source-id field against `SourceRagService` (rag_controller.dart uses `_activeCollection.addDocumentUtf8(...)`). Adjust the two flagged lines to the real signature — no other logic changes. + +- [ ] **Step 2: Commit** +```bash +git add example/lib/profiling/query_fixture.dart +git commit -m "feat(profiling): deterministic A/B fixture + query lanes (LOC-67)" +``` + +## Task P2.3: integration_test entrypoint skeleton (engine init + fixture + smoke) + +**Files:** Create `example/integration_test/query_profile_test.dart` + +- [ ] **Step 1: Implement skeleton** +```dart +// example/integration_test/query_profile_test.dart +import 'package:flutter_test/flutter_test.dart'; +import 'package:integration_test/integration_test.dart'; +import 'package:mobile_rag_engine/mobile_rag_engine.dart'; +import 'package:rag_engine_example/profiling/query_fixture.dart'; + +void main() { + IntegrationTestWidgetsFlutterBinding.ensureInitialized(); + + const docsPerCollection = 500; // representative; raise for scan-bound runs + + testWidgets('profiler: engine init + fixture builds', (tester) async { + await MobileRag.initialize( + tokenizerAsset: 'assets/tokenizer.json', + modelAsset: 'assets/model.onnx', + databaseName: 'profile.sqlite', + deferIndexWarmup: true, + ); + final ids = await QueryFixture.seed(docsPerCollection: docsPerCollection); + expect(ids[QueryFixture.collectionA]!.length, docsPerCollection); + expect(ids[QueryFixture.collectionB]!.length, docsPerCollection); + }); +} +``` + +- [ ] **Step 2: Run on the connected device** +Run: `cd example && flutter test integration_test/query_profile_test.dart -d ` +Expected: PASS (fixture builds on device). `flutter devices` to get ``. + +- [ ] **Step 3: Commit** +```bash +git add example/integration_test/query_profile_test.dart +git commit -m "test(profiling): integration_test entrypoint + fixture smoke (LOC-67)" +``` + +> **PR P2 ends here.** Open PR, fill `PR-P2.md` + README, user merges. + +--- + +## Task P3.1: Segment-timing driver (embed/activate/search/hydrate + metrics) + +**Files:** Create `example/lib/profiling/query_profiler.dart` + +Segments are measured by replicating the steps the high-level `searchHybrid` bundles, so each is isolated: +`embed` = `EmbeddingService.embed`; `activate` = `activateCollectionForHybridSearch`; `search` = raw `searchMetaHybrid` (no embed/activate inside it — verified); `hydrate` = `handle.hydrateChunks`. + +- [ ] **Step 1: Implement** +```dart +// example/lib/profiling/query_profiler.dart +import 'package:mobile_rag_engine/services/embedding_service.dart'; +import 'package:mobile_rag_engine/src/rust/api/source_rag.dart' as rust_rag; +import 'package:mobile_rag_engine/src/rust/api/query_metrics.dart' as qm; +import 'package:mobile_rag_engine/services/source_rag_service.dart' show kDefaultVectorWeight, kDefaultBm25Weight; +import 'query_profile_report.dart'; + +class QueryProfiler { + final String indexBasePath; // _indexPath used by activate (confirm via SourceRagService._indexPath getter or MobileRag) + QueryProfiler(this.indexBasePath); + + static Future _timeMs(Future Function() fn) async { + final sw = Stopwatch()..start(); + await fn(); + return sw.elapsedMicroseconds / 1000.0; + } + + /// One measured query. If [activateCollection] != null, the activate segment + /// is measured (switching/cold). Returns segment ms keyed by name. + Future> measureOnce({ + required String collectionId, + required String query, + List? sourceIds, // non-null => filtered lane (i8 exact-scan) + String? activateCollection, // non-null => measure activate (switch) + int topK = 10, + }) async { + final out = {}; + + if (activateCollection != null) { + out['activate'] = await _timeMs(() => rust_rag.activateCollectionForHybridSearch( + collectionId: activateCollection, basePath: indexBasePath)); + } + + late final dynamic embedding; + out['embed'] = await _timeMs(() async { embedding = await EmbeddingService.embed(query); }); + + qm.resetQueryContentReadStats(); + + late final rust_rag.SearchHandle handle; + out['search'] = await _timeMs(() async { + handle = await rust_rag.searchMetaHybrid( + collectionId: collectionId, + queryText: query, + queryEmbedding: embedding, + options: rust_rag.SearchMetaHybridOptions( + topK: topK, + vectorWeight: kDefaultVectorWeight, + bm25Weight: kDefaultBm25Weight, + sourceIds: sourceIds == null ? null : _toI64(sourceIds), + adjacentChunks: 0, + ), + ); + }); + + final hits = await handle.hitMeta(); + final chunkIds = _toI64(hits.map((h) => h.chunkId as int).toList()); // confirm hit field name + out['hydrate'] = await _timeMs(() async { await handle.hydrateChunks(chunkIds: chunkIds); }); + + await handle.dispose(); + return out; + } + + // dart:typed_data Int64List builder; confirm _toInt64List equivalent exists or inline. + static dynamic _toI64(List v) => /* Int64List.fromList(v) */ throw UnimplementedError(); +} +``` +> Execution notes (confirm against signatures, adjust only flagged lines): `indexBasePath` source (`SourceRagService` builds `_indexPath` from the docs dir + collection — reuse that construction); the hit's chunk-id field name (`SearchHitMeta`); replace `_toI64` with `Int64List.fromList` (import `dart:typed_data`). No control-flow changes. + +- [ ] **Step 2: Snapshot helper for query_metrics → io map** +```dart +Map snapshotIo() { + final s = qm.takeQueryContentReadStats(); + return { + 'scoped_exact_scan_rows': s.scopedExactScanRows.toInt(), + 'scoped_exact_scan_content_bytes': s.scopedExactScanContentBytes.toInt(), + 'scoped_exact_scan_tokens': s.scopedExactScanTokens.toInt(), + 'scoped_exact_scan_tokenization_nanos': s.scopedExactScanTokenizationNanos.toInt(), + 'full_hydrate_rows': s.fullHydrateRows.toInt(), + 'full_hydrate_content_bytes': s.fullHydrateContentBytes.toInt(), + }; +} +``` +- [ ] **Step 3: Commit** +```bash +git add example/lib/profiling/query_profiler.dart +git commit -m "feat(profiling): per-segment query profiler driver (LOC-68)" +``` + +## Task P3.2: Scenario runner — Pure Cold / Switching Cold / Pure Warm × 2 lanes + +**Files:** Modify `example/integration_test/query_profile_test.dart` + +- [ ] **Step 1: Implement scenarios** (append a second `testWidgets`) +```dart +testWidgets('profiler: scenarios x lanes', (tester) async { + await MobileRag.initialize( + tokenizerAsset: 'assets/tokenizer.json', modelAsset: 'assets/model.onnx', + databaseName: 'profile.sqlite', deferIndexWarmup: true); + final ids = await QueryFixture.seed(docsPerCollection: 500); + final profiler = QueryProfiler(/* indexBasePath */); + final runs = []; + const measured = 30, warmup = 5; + + Future runCell({required String lane, required String category, + List? sourceIds, bool measureActivate = false}) async { + final segs = { + for (final n in ['embed','search','hydrate', if (measureActivate) 'activate']) + n: SegmentSamples(n), + }; + final q = QueryFixture.unfilteredQueries; + // warmup (discard) + for (var i = 0; i < warmup; i++) { + await profiler.measureOnce(collectionId: QueryFixture.collectionA, + query: q[i % q.length], sourceIds: sourceIds); + } + for (var i = 0; i < measured; i++) { + // switching_cold: re-activate A each iter after touching B; else activate once/never + final actCol = measureActivate ? QueryFixture.collectionA : null; + if (measureActivate) { + await profiler.measureOnce(collectionId: QueryFixture.collectionB, + query: q[0], activateCollection: QueryFixture.collectionB); // touch B → A is now cold + } + final m = await profiler.measureOnce(collectionId: QueryFixture.collectionA, + query: q[i % q.length], sourceIds: sourceIds, activateCollection: actCol); + m.forEach((k, v) => segs[k]?.add(v)); + } + runs.add(QueryProfileRun(lane: lane, category: category, + segments: segs, io: profiler.snapshotIo(), meta: const {})); + } + + // Pure Warm (no activate between iters) — both lanes + await runCell(lane: 'unfiltered', category: 'pure_warm'); + await runCell(lane: 'filtered', category: 'pure_warm', + sourceIds: ids[QueryFixture.collectionA]!.take(3).toList()); + // Switching Cold (A→B→A each iter) — unfiltered (the expensive path) + await runCell(lane: 'unfiltered', category: 'switching_cold', measureActivate: true); + + // sanity: every cell has >=1 sample per segment (fail-closed) + for (final r in runs) { + for (final s in r.segments.values) { + expect(s.samplesMs.length, greaterThan(0), reason: '${r.lane}/${r.category}/${s.segment}'); + } + } + // exported in P4 +}); +``` +> Pure Cold (app first launch) is a separate one-shot: capture the very first `measureOnce` before any warmup in a dedicated `testWidgets` that does NOT pre-warm. Add as `category: 'pure_cold'` with `measuredIterations: 1`. + +- [ ] **Step 2: Run on device** +Run: `cd example && flutter test integration_test/query_profile_test.dart -d ` +Expected: PASS; logs show per-segment counts. Inspect the printed numbers. + +- [ ] **Step 3: Commit** +```bash +git add example/integration_test/query_profile_test.dart +git commit -m "test(profiling): cold/switch/warm scenarios x 2 lanes (LOC-68)" +``` + +> **PR P3 ends here.** Open PR, fill `PR-P3.md` (paste observed per-segment p50/p95!) + README, user merges. + +--- + +## Task P4.1: Export JSON+CSV to app docs dir + structured log + metadata + +**Files:** Modify `example/integration_test/query_profile_test.dart`; add `example/lib/profiling/profile_export.dart` + +- [ ] **Step 1: Implement exporter** +```dart +// example/lib/profiling/profile_export.dart +import 'dart:io'; +import 'package:path_provider/path_provider.dart'; +import 'query_profile_report.dart'; + +class ProfileExport { + /// Writes /query_profile.json and .csv; returns the dir path. + static Future write(QueryProfileReport report, {required String tsTag}) async { + final dir = await getApplicationDocumentsDirectory(); + final base = '${dir.path}/query_profile_$tsTag'; + await File('$base.json').writeAsString(report.toJsonString(), flush: true); + await File('$base.csv').writeAsString(report.toCsv(), flush: true); + // structured one-line log per run for live logcat/console monitoring + for (final r in report.runs) { + // ignore: avoid_print + print('PROFILE ${r.toJson()}'); + } + return dir.path; + } +} +``` +- [ ] **Step 2: Wire into the scenario test** (after building `runs`) +```dart +final report = QueryProfileReport(runs: runs); +final ts = DateTime.now().millisecondsSinceEpoch.toString(); +final path = await ProfileExport.write(report, tsTag: ts); +print('PROFILE_EXPORT_DIR $path'); // adb pull / xcrun simctl from here +``` +> Attach run metadata in `meta`: device model, OS, corpus size, top_k, feature flags (release ships `vector_faer,vector_quant_i8`), charging state (document manually — In scope per DESIGN §10). Populate `meta` map in `runCell`. + +- [ ] **Step 3: Run on device + pull** +Run: `flutter test integration_test/query_profile_test.dart -d ` +Then Android: `adb shell run-as cat files/query_profile_.json > out.json` (or `adb pull` from app docs); iOS: `xcrun simctl get_app_container booted data` then read Documents. +Expected: JSON+CSV present, one `PROFILE` log line per run. + +- [ ] **Step 4: Commit** +```bash +git add example/lib/profiling/profile_export.dart example/integration_test/query_profile_test.dart +git commit -m "feat(profiling): JSON/CSV export + structured log + metadata (LOC-69)" +``` + +> **PR P4 ends here.** This is the **baseline deliverable** — open PR, fill `PR-P4.md` with the pulled baseline numbers + which segment dominates, README, user merges. + +--- + +## Phase 2 (PR P5, CONDITIONAL — gated on P3/P4 data) + +Decision rule from the baseline (do NOT pre-build): +- **embed dominates** → no Rust work. Conclusion: next target is ONNX inference (out of this crate). Record in `PR-P5.md` / RETRO; close initiative. +- **activate dominates (cold/switch)** → split `activate` into `bm25_rebuild` vs `hnsw_load`: add timing around `rebuild_chunk_bm25_index_for_collection` vs `load_hnsw_index` (Rust log timestamps or a small `QueryTimings`-style snapshot mirroring `query_metrics.rs`). **Mandatory for cold/switch.** +- **search dominates (warm)** → add Rust `QueryTimings` (thread-local `Instant` stage timers in the hybrid path, FFI snapshot like `query_metrics.rs`) splitting ANN / BM25-rank / RRF; surface via the same export. +- **hydrate/IO dominates** → already covered by `query_metrics` counters; just chart them. + +--- + +## Self-Review (run against DESIGN.md) + +**Spec coverage:** embed/activate/search/hydrate segments → P3.1 ✅; FFI-overhead formula → P1.2 `ffiOverheadMs` ✅; 2 lanes → P3.2 (`sourceIds`) ✅; Cold/Switching/Warm → P3.2 + Pure-Cold one-shot ✅; A→B→A switch → P3.2 `measureActivate` ✅; query_metrics I/O → P3.1 `snapshotIo` ✅; JSON/CSV+log+metadata → P4.1 ✅; reuse benchmark_service → P1.1 ✅; fail-closed N>0 → P3.2 ✅; example-app/real-ONNX → P2.3 ✅; Phase-2 gate → P5 ✅. Out-of-scope (indexing/memory/battery/CI-gate/recall) correctly absent. + +**Placeholder scan:** Remaining `confirm ...` notes are execution-time signature confirmations on **flagged single lines** (addDocumentUtf8 arg, source-id field, hit chunk-id field, `_toI64`→`Int64List.fromList`, `indexBasePath` source), each with the exact reference to confirm against — not vague requirements. All logic/control-flow is concrete. + +**Type consistency:** `SegmentSamples`/`QueryProfileRun`/`QueryProfileReport` used consistently P1.2→P3→P4; `statsFromSamples` signature matches P1.1; `DetailedBenchmarkStats.toJson` field names match benchmark_models.dart. diff --git a/docs/perf/ondevice-query-profiler/README.md b/docs/perf/ondevice-query-profiler/README.md index 87a5ab4..ff877fc 100644 --- a/docs/perf/ondevice-query-profiler/README.md +++ b/docs/perf/ondevice-query-profiler/README.md @@ -6,7 +6,8 @@ vector_math 커널 슬라이스는 거의 최적임을 확인했으나 **온디바이스 RAG end-to-end는 미측정**. 실기에서 쿼리 단계별 지연을 분해해 진짜 병목을 데이터로 찾는다. ## 문서 -- [DESIGN.md](DESIGN.md) — 승인된 설계 스펙(검증된 아키텍처 사실 포함). 구현 계획은 writing-plans 산출 후 추가. +- [DESIGN.md](DESIGN.md) — 승인된 설계 스펙(검증된 아키텍처 사실 포함). +- [PLAN.md](PLAN.md) — 구현 계획(PR P1–P5, TDD 단계·검증된 시그니처). - Linear 프로젝트: [온디바이스 RAG 쿼리 프로파일러](https://linear.app/loceract/project/온디바이스-rag-쿼리-프로파일러-25df240c4262) · 설계 이슈 [LOC-65](https://linear.app/loceract/issue/LOC-65) ## 핵심 설계 결론 (요약) @@ -20,7 +21,12 @@ vector_math 커널 슬라이스는 거의 최적임을 확인했으나 **온디 | PR | 제목 | Linear | 상태 | |----|------|--------|------| -| — | 설계 스펙 (이 문서) | [LOC-65](https://linear.app/loceract/issue/LOC-65) | 🟦 진행 | +| 스펙+계획 | DESIGN + PLAN (이 PR) | [LOC-65](https://linear.app/loceract/issue/LOC-65) | 🟦 진행 | +| P1 | stats + report 모델 + export utils (host-TDD) | [LOC-66](https://linear.app/loceract/issue/LOC-66) | ⬜ TODO | +| P2 | example integration_test 배선 + A/B 픽스처 | [LOC-67](https://linear.app/loceract/issue/LOC-67) | ⬜ TODO | +| P3 | 세그먼트 타이밍 + 3시나리오 + metrics 스냅샷 | [LOC-68](https://linear.app/loceract/issue/LOC-68) | ⬜ TODO | +| P4 | JSON/CSV export + 로그 + 메타 (baseline 산출) | [LOC-69](https://linear.app/loceract/issue/LOC-69) | ⬜ TODO | +| P5 | (조건부) Phase-2 드릴다운 — 지배 버킷별 | [LOC-70](https://linear.app/loceract/issue/LOC-70) | ⏸ 데이터 게이트 | ## 규약 (프로젝트 공통) - CI: `cargo test -- --test-threads=1`. 커밋/PR에 Claude 귀속 미포함. PR은 열고 CI green까지만, 머지는 본인.