diff --git a/docs/perf/ondevice-query-profiler/PR-P1.md b/docs/perf/ondevice-query-profiler/PR-P1.md new file mode 100644 index 0000000..3a3cc2d --- /dev/null +++ b/docs/perf/ondevice-query-profiler/PR-P1.md @@ -0,0 +1,23 @@ +# PR P1 — 프로파일러 report 모델 + JSON/CSV (host-TDD) + +- 브랜치: `feat/loc-66-profiler-stats-report` +- PR: [#70](https://github.com/dev07060/mobile_rag_engine/pull/70) · Linear [LOC-66](https://linear.app/loceract/issue/LOC-66) +- 상태: 🟩 머지 + +## 스코프 (무엇을/왜) +프로파일러 결과 모델 + 직렬화(순수 로직, 기기 불필요). subagent-driven으로 실행(구현→spec→quality 2단계 리뷰). +- `example/lib/profiling/query_profile_report.dart`: `SegmentSamples` / `QueryProfileRun` / `QueryProfileReport` — 세그먼트 샘플을 **기존 `BenchmarkService.summarizeSamples`로 집계**, JSON/CSV 직렬화, `ffiOverheadMs`(Dart−Rust, clamp 0). +- `example/test/profiling/query_profile_report_test.dart`: 4 테스트. + +## 결과 (Before → After) +- 호스트 테스트 **4/4 green**. CI 6/6 green(#70). +- spec-compliance ✅ / code-quality ✅(analyzer clean). + +## 받은 피드백 (리뷰) +- code-quality minor 반영: CSV `n`=`s.measuredIterations`(일관성), ffiOverhead 테스트 허용오차 1e-9→1e-6, empty-segment 테스트 추가. + +## 결정 로그 (계획 대비 변경) +- **원안 P1.1 `BenchmarkService.statsFromSamples` 폐기 (DRY).** 구현 착수 후 기존 `summarizeSamples(samples, warmupIterations:0)`가 동일 기능임을 발견(초기 Explore가 놓침) → 중복 메서드 추가 대신 기존 것 재사용. "구현 중 검증"이 중복을 잡은 사례. + +## 리스크 / 롤백 +- 순수 로직·추가만. 롤백: PR revert. 동작 코드 영향 없음. diff --git a/docs/perf/ondevice-query-profiler/PR-P2.md b/docs/perf/ondevice-query-profiler/PR-P2.md new file mode 100644 index 0000000..30755cd --- /dev/null +++ b/docs/perf/ondevice-query-profiler/PR-P2.md @@ -0,0 +1,25 @@ +# PR P2 — example integration_test 배선 + A/B 픽스처 + +- 브랜치: `feat/loc-67-profiler-fixture` +- Linear: [LOC-67](https://linear.app/loceract/issue/LOC-67) +- 상태: 🟦 진행 (PR 열림, 기기 스모크 green) + +## 스코프 +실 ONNX 자산이 있는 example 앱에서 구동하는 디바이스 프로파일러의 기반. +- `example/pubspec.yaml`: `integration_test` dev-dep. +- `example/lib/profiling/query_fixture.dart`: 결정적 A/B 코퍼스(`MobileRag.inCollection(id).addDocumentUtf8`) + 2레인 쿼리셋. +- `example/integration_test/query_profile_test.dart`: 엔진 init(실 ONNX) + 픽스처 시드 스모크. +- (P1 저널 동기화 a3f35f7 포함: PR-P1.md + README 상태표.) + +## 결과 (실기 검증, iPhone iOS 26.5) +- **`flutter test integration_test/query_profile_test.dart -d ` → `+1: All tests passed!`** +- 엔진 init(실 ONNX 모델 107MB) + 40×2 = 80 소스 시드 + assertion 통과. + +## 받은 피드백 / 디버그 로그 +- 1차: `testWidgets` 의 end-of-test `SemanticsHandle` 검증 실패(본문은 정상 실행). → **비-UI 본문이라 `test()` 로 교체**(위젯-테스터 검증 회피). 검증된 픽스. +- 2차: 물리 iOS에서 `Dart VM Service not discovered`(DDS) — macOS **Xcode 자동화 권한** 미허가 + stale 프로세스. → 권한 허가 + stale 종료 후 재실행 green. + +## 리스크 / 롤백 / 다음 +- 동작 코드 변경 없음(테스트/픽스처/스크립트/dep). 롤백: PR revert. +- ⚠️ **P3/P4는 profile 모드 필수** — `flutter test integration_test`는 기본 debug → cargokit debug → fallback 백엔드. 출시 `vector_faer,vector_quant_i8` 측정하려면 profile 모드로 실행해야 함(P3 진입 시 invocation 확정). +- ⚠️ 물리 iOS 구동 전제: Xcode 자동화 권한 + 기기 unlocked. (CI 비포함 — 로컬 기기 측정 전용) diff --git a/docs/perf/ondevice-query-profiler/README.md b/docs/perf/ondevice-query-profiler/README.md index ff877fc..da8703d 100644 --- a/docs/perf/ondevice-query-profiler/README.md +++ b/docs/perf/ondevice-query-profiler/README.md @@ -21,9 +21,9 @@ vector_math 커널 슬라이스는 거의 최적임을 확인했으나 **온디 | PR | 제목 | Linear | 상태 | |----|------|--------|------| -| 스펙+계획 | DESIGN + PLAN (이 PR) | [LOC-65](https://linear.app/loceract/issue/LOC-65) | 🟦 진행 | -| P1 | stats + report 모델 + export utils (host-TDD) | [LOC-66](https://linear.app/loceract/issue/LOC-66) | ⬜ TODO | -| P2 | example integration_test 배선 + A/B 픽스처 | [LOC-67](https://linear.app/loceract/issue/LOC-67) | ⬜ TODO | +| 스펙+계획 | DESIGN + PLAN | [LOC-65](https://linear.app/loceract/issue/LOC-65) | 🟩 머지(#69) | +| P1 | report 모델 + JSON/CSV (host-TDD) | [LOC-66](https://linear.app/loceract/issue/LOC-66) | 🟩 머지(#70, [PR-P1.md](PR-P1.md)) | +| P2 | example integration_test 배선 + A/B 픽스처 | [LOC-67](https://linear.app/loceract/issue/LOC-67) | 🟦 진행([PR-P2.md](PR-P2.md), 기기 green) | | P3 | 세그먼트 타이밍 + 3시나리오 + metrics 스냅샷 | [LOC-68](https://linear.app/loceract/issue/LOC-68) | ⬜ TODO | | P4 | JSON/CSV export + 로그 + 메타 (baseline 산출) | [LOC-69](https://linear.app/loceract/issue/LOC-69) | ⬜ TODO | | P5 | (조건부) Phase-2 드릴다운 — 지배 버킷별 | [LOC-70](https://linear.app/loceract/issue/LOC-70) | ⏸ 데이터 게이트 | diff --git a/example/integration_test/query_profile_test.dart b/example/integration_test/query_profile_test.dart new file mode 100644 index 0000000..6416116 --- /dev/null +++ b/example/integration_test/query_profile_test.dart @@ -0,0 +1,34 @@ +import 'package:flutter_test/flutter_test.dart'; +import 'package:integration_test/integration_test.dart'; +import 'package:mobile_rag_engine/mobile_rag_engine.dart'; +import 'package:mobile_rag_engine_example/profiling/query_fixture.dart'; + +// On-device RAG query profiler — P2.3 smoke entrypoint. +// +// Validates the device pipeline: engine init (real ONNX), A/B fixture seed. +// NOTE: `flutter test integration_test` builds DEBUG by default, which uses +// the cargokit debug profile (fallback Rust backend). The P3/P4 measurement +// runs must use profile/release so the shipped `vector_faer,vector_quant_i8` +// backend is exercised. +void main() { + IntegrationTestWidgetsFlutterBinding.ensureInitialized(); + + // Smoke uses a small corpus for speed; P3 scales this up. + const docsPerCollection = 40; + + // Non-UI integration test: use test() (not testWidgets) so the widget-tester + // end-of-test semantics-handle verification — irrelevant here — doesn't fire. + test('profiler smoke: engine init + A/B fixture builds', () async { + await MobileRag.initialize( + tokenizerAsset: 'assets/tokenizer.json', + modelAsset: 'assets/model.onnx', + databaseName: 'profile.sqlite', + deferIndexWarmup: true, + ); + + final ids = await QueryFixture.seed(docsPerCollection: docsPerCollection); + + expect(ids[QueryFixture.collectionA]!.length, docsPerCollection); + expect(ids[QueryFixture.collectionB]!.length, docsPerCollection); + }); +} diff --git a/example/lib/profiling/query_fixture.dart b/example/lib/profiling/query_fixture.dart new file mode 100644 index 0000000..5799795 --- /dev/null +++ b/example/lib/profiling/query_fixture.dart @@ -0,0 +1,47 @@ +import 'dart:convert'; +import 'dart:typed_data'; +import 'package:mobile_rag_engine/mobile_rag_engine.dart'; + +/// Deterministic fixture: two collections (A/B) with reproducible short docs, +/// plus a fixed query set. Used by the on-device query profiler integration test. +class QueryFixture { + static const collectionA = 'profile_a'; + static const collectionB = 'profile_b'; + + /// Deterministic short docs for a collection. `count` controls corpus size. + static List docs(String seedTag, int count) => List.generate( + count, + (i) => 'doc $seedTag $i: mobile retrieval augmented generation ' + 'embedding vector search bm25 ranking topic${i % 17} ' + 'token${(i * 7) % 53} alpha beta gamma delta epsilon', + ); + + /// Lane-1 (unfiltered) query set. Lane-2 (filtered) injects sourceIds at call time. + static const unfilteredQueries = [ + 'vector search ranking', + 'embedding topic3 retrieval', + 'bm25 token alpha', + 'mobile generation gamma', + 'topic9 delta epsilon', + ]; + + /// Seed both collections via the real ingest path (chunks + embeddings + indexes). + /// Returns source ids per collection (used to drive the filtered lane). + static Future>> seed({required int docsPerCollection}) async { + final ids = >{}; + for (final c in [collectionA, collectionB]) { + final col = MobileRag.instance.inCollection(c); + final ds = docs(c, docsPerCollection); + final srcIds = []; + for (var i = 0; i < ds.length; i++) { + final r = await col.addDocumentUtf8( + Uint8List.fromList(utf8.encode(ds[i])), + name: 'doc_${c}_$i', + ); + srcIds.add(r.sourceId); + } + ids[c] = srcIds; + } + return ids; + } +} diff --git a/example/pubspec.lock b/example/pubspec.lock index c758c58..584b438 100644 --- a/example/pubspec.lock +++ b/example/pubspec.lock @@ -81,6 +81,14 @@ packages: url: "https://pub.dev" source: hosted version: "2.1.5" + file: + dependency: transitive + description: + name: file + sha256: a3b4f84adafef897088c160faf7dfffb7696046cb13ae90b508c2cbc95d3b8d4 + url: "https://pub.dev" + source: hosted + version: "7.0.1" file_picker: dependency: "direct main" description: @@ -94,6 +102,11 @@ packages: description: flutter source: sdk version: "0.0.0" + flutter_driver: + dependency: transitive + description: flutter + source: sdk + version: "0.0.0" flutter_lints: dependency: "direct dev" description: @@ -136,6 +149,16 @@ packages: url: "https://pub.dev" source: hosted version: "3.1.0" + fuchsia_remote_debug_protocol: + dependency: transitive + description: flutter + source: sdk + version: "0.0.0" + integration_test: + dependency: "direct dev" + description: flutter + source: sdk + version: "0.0.0" json_annotation: dependency: transitive description: @@ -287,6 +310,14 @@ packages: url: "https://pub.dev" source: hosted version: "2.1.8" + process: + dependency: transitive + description: + name: process + sha256: c6248e4526673988586e8c00bb22a49210c258dc91df5227d5da9748ecf79744 + url: "https://pub.dev" + source: hosted + version: "5.0.5" rag_engine_flutter: dependency: "direct overridden" description: @@ -331,6 +362,14 @@ packages: url: "https://pub.dev" source: hosted version: "1.4.1" + sync_http: + dependency: transitive + description: + name: sync_http + sha256: "7f0cd72eca000d2e026bcd6f990b81d0ca06022ef4e32fb257b30d3d1014a961" + url: "https://pub.dev" + source: hosted + version: "0.3.1" term_glyph: dependency: transitive description: @@ -371,6 +410,14 @@ packages: url: "https://pub.dev" source: hosted version: "1.1.1" + webdriver: + dependency: transitive + description: + name: webdriver + sha256: "2f3a14ca026957870cfd9c635b83507e0e51d8091568e90129fbf805aba7cade" + url: "https://pub.dev" + source: hosted + version: "3.1.0" win32: dependency: transitive description: diff --git a/example/pubspec.yaml b/example/pubspec.yaml index 241c2ba..93cbf97 100644 --- a/example/pubspec.yaml +++ b/example/pubspec.yaml @@ -18,6 +18,8 @@ dev_dependencies: flutter_test: sdk: flutter flutter_lints: ^5.0.0 + integration_test: + sdk: flutter dependency_overrides: rag_engine_flutter: