Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions docs/perf/ondevice-query-profiler/PR-P1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# PR P1 — 프로파일러 report 모델 + JSON/CSV (host-TDD)

- 브랜치: `feat/loc-66-profiler-stats-report`
- PR: [#70](https://github.com/dev07060/mobile_rag_engine/pull/70) · Linear [LOC-66](https://linear.app/loceract/issue/LOC-66)
- 상태: 🟩 머지

## 스코프 (무엇을/왜)
프로파일러 결과 모델 + 직렬화(순수 로직, 기기 불필요). subagent-driven으로 실행(구현→spec→quality 2단계 리뷰).
- `example/lib/profiling/query_profile_report.dart`: `SegmentSamples` / `QueryProfileRun` / `QueryProfileReport` — 세그먼트 샘플을 **기존 `BenchmarkService.summarizeSamples`로 집계**, JSON/CSV 직렬화, `ffiOverheadMs`(Dart−Rust, clamp 0).
- `example/test/profiling/query_profile_report_test.dart`: 4 테스트.

## 결과 (Before → After)
- 호스트 테스트 **4/4 green**. CI 6/6 green(#70).
- spec-compliance ✅ / code-quality ✅(analyzer clean).

## 받은 피드백 (리뷰)
- code-quality minor 반영: CSV `n`=`s.measuredIterations`(일관성), ffiOverhead 테스트 허용오차 1e-9→1e-6, empty-segment 테스트 추가.

## 결정 로그 (계획 대비 변경)
- **원안 P1.1 `BenchmarkService.statsFromSamples` 폐기 (DRY).** 구현 착수 후 기존 `summarizeSamples(samples, warmupIterations:0)`가 동일 기능임을 발견(초기 Explore가 놓침) → 중복 메서드 추가 대신 기존 것 재사용. "구현 중 검증"이 중복을 잡은 사례.

## 리스크 / 롤백
- 순수 로직·추가만. 롤백: PR revert. 동작 코드 영향 없음.
25 changes: 25 additions & 0 deletions docs/perf/ondevice-query-profiler/PR-P2.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# PR P2 — example integration_test 배선 + A/B 픽스처

- 브랜치: `feat/loc-67-profiler-fixture`
- Linear: [LOC-67](https://linear.app/loceract/issue/LOC-67)
- 상태: 🟦 진행 (PR 열림, 기기 스모크 green)

## 스코프
실 ONNX 자산이 있는 example 앱에서 구동하는 디바이스 프로파일러의 기반.
- `example/pubspec.yaml`: `integration_test` dev-dep.
- `example/lib/profiling/query_fixture.dart`: 결정적 A/B 코퍼스(`MobileRag.inCollection(id).addDocumentUtf8`) + 2레인 쿼리셋.
- `example/integration_test/query_profile_test.dart`: 엔진 init(실 ONNX) + 픽스처 시드 스모크.
- (P1 저널 동기화 a3f35f7 포함: PR-P1.md + README 상태표.)

## 결과 (실기 검증, iPhone iOS 26.5)
- **`flutter test integration_test/query_profile_test.dart -d <iphone>` → `+1: All tests passed!`**
- 엔진 init(실 ONNX 모델 107MB) + 40×2 = 80 소스 시드 + assertion 통과.

## 받은 피드백 / 디버그 로그
- 1차: `testWidgets` 의 end-of-test `SemanticsHandle` 검증 실패(본문은 정상 실행). → **비-UI 본문이라 `test()` 로 교체**(위젯-테스터 검증 회피). 검증된 픽스.
- 2차: 물리 iOS에서 `Dart VM Service not discovered`(DDS) — macOS **Xcode 자동화 권한** 미허가 + stale 프로세스. → 권한 허가 + stale 종료 후 재실행 green.

## 리스크 / 롤백 / 다음
- 동작 코드 변경 없음(테스트/픽스처/스크립트/dep). 롤백: PR revert.
- ⚠️ **P3/P4는 profile 모드 필수** — `flutter test integration_test`는 기본 debug → cargokit debug → fallback 백엔드. 출시 `vector_faer,vector_quant_i8` 측정하려면 profile 모드로 실행해야 함(P3 진입 시 invocation 확정).
- ⚠️ 물리 iOS 구동 전제: Xcode 자동화 권한 + 기기 unlocked. (CI 비포함 — 로컬 기기 측정 전용)
6 changes: 3 additions & 3 deletions docs/perf/ondevice-query-profiler/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,9 @@ vector_math 커널 슬라이스는 거의 최적임을 확인했으나 **온디

| PR | 제목 | Linear | 상태 |
|----|------|--------|------|
| 스펙+계획 | DESIGN + PLAN (이 PR) | [LOC-65](https://linear.app/loceract/issue/LOC-65) | 🟦 진행 |
| P1 | stats + report 모델 + export utils (host-TDD) | [LOC-66](https://linear.app/loceract/issue/LOC-66) | ⬜ TODO |
| P2 | example integration_test 배선 + A/B 픽스처 | [LOC-67](https://linear.app/loceract/issue/LOC-67) | ⬜ TODO |
| 스펙+계획 | DESIGN + PLAN | [LOC-65](https://linear.app/loceract/issue/LOC-65) | 🟩 머지(#69) |
| P1 | report 모델 + JSON/CSV (host-TDD) | [LOC-66](https://linear.app/loceract/issue/LOC-66) | 🟩 머지(#70, [PR-P1.md](PR-P1.md)) |
| P2 | example integration_test 배선 + A/B 픽스처 | [LOC-67](https://linear.app/loceract/issue/LOC-67) | 🟦 진행([PR-P2.md](PR-P2.md), 기기 green) |
| P3 | 세그먼트 타이밍 + 3시나리오 + metrics 스냅샷 | [LOC-68](https://linear.app/loceract/issue/LOC-68) | ⬜ TODO |
| P4 | JSON/CSV export + 로그 + 메타 (baseline 산출) | [LOC-69](https://linear.app/loceract/issue/LOC-69) | ⬜ TODO |
| P5 | (조건부) Phase-2 드릴다운 — 지배 버킷별 | [LOC-70](https://linear.app/loceract/issue/LOC-70) | ⏸ 데이터 게이트 |
Expand Down
34 changes: 34 additions & 0 deletions example/integration_test/query_profile_test.dart
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
import 'package:flutter_test/flutter_test.dart';
import 'package:integration_test/integration_test.dart';
import 'package:mobile_rag_engine/mobile_rag_engine.dart';
import 'package:mobile_rag_engine_example/profiling/query_fixture.dart';

// On-device RAG query profiler — P2.3 smoke entrypoint.
//
// Validates the device pipeline: engine init (real ONNX), A/B fixture seed.
// NOTE: `flutter test integration_test` builds DEBUG by default, which uses
// the cargokit debug profile (fallback Rust backend). The P3/P4 measurement
// runs must use profile/release so the shipped `vector_faer,vector_quant_i8`
// backend is exercised.
void main() {
IntegrationTestWidgetsFlutterBinding.ensureInitialized();

// Smoke uses a small corpus for speed; P3 scales this up.
const docsPerCollection = 40;

// Non-UI integration test: use test() (not testWidgets) so the widget-tester
// end-of-test semantics-handle verification — irrelevant here — doesn't fire.
test('profiler smoke: engine init + A/B fixture builds', () async {
await MobileRag.initialize(
tokenizerAsset: 'assets/tokenizer.json',
modelAsset: 'assets/model.onnx',
databaseName: 'profile.sqlite',
deferIndexWarmup: true,
);

final ids = await QueryFixture.seed(docsPerCollection: docsPerCollection);

expect(ids[QueryFixture.collectionA]!.length, docsPerCollection);
expect(ids[QueryFixture.collectionB]!.length, docsPerCollection);
});
}
47 changes: 47 additions & 0 deletions example/lib/profiling/query_fixture.dart
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
import 'dart:convert';
import 'dart:typed_data';
import 'package:mobile_rag_engine/mobile_rag_engine.dart';

/// Deterministic fixture: two collections (A/B) with reproducible short docs,
/// plus a fixed query set. Used by the on-device query profiler integration test.
class QueryFixture {
static const collectionA = 'profile_a';
static const collectionB = 'profile_b';

/// Deterministic short docs for a collection. `count` controls corpus size.
static List<String> docs(String seedTag, int count) => List.generate(
count,
(i) => 'doc $seedTag $i: mobile retrieval augmented generation '
'embedding vector search bm25 ranking topic${i % 17} '
'token${(i * 7) % 53} alpha beta gamma delta epsilon',
);

/// Lane-1 (unfiltered) query set. Lane-2 (filtered) injects sourceIds at call time.
static const unfilteredQueries = <String>[
'vector search ranking',
'embedding topic3 retrieval',
'bm25 token alpha',
'mobile generation gamma',
'topic9 delta epsilon',
];

/// Seed both collections via the real ingest path (chunks + embeddings + indexes).
/// Returns source ids per collection (used to drive the filtered lane).
static Future<Map<String, List<int>>> seed({required int docsPerCollection}) async {
final ids = <String, List<int>>{};
for (final c in [collectionA, collectionB]) {
final col = MobileRag.instance.inCollection(c);
final ds = docs(c, docsPerCollection);
final srcIds = <int>[];
for (var i = 0; i < ds.length; i++) {
final r = await col.addDocumentUtf8(
Uint8List.fromList(utf8.encode(ds[i])),
name: 'doc_${c}_$i',
);
srcIds.add(r.sourceId);
}
ids[c] = srcIds;
}
return ids;
}
}
47 changes: 47 additions & 0 deletions example/pubspec.lock
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,14 @@ packages:
url: "https://pub.dev"
source: hosted
version: "2.1.5"
file:
dependency: transitive
description:
name: file
sha256: a3b4f84adafef897088c160faf7dfffb7696046cb13ae90b508c2cbc95d3b8d4
url: "https://pub.dev"
source: hosted
version: "7.0.1"
file_picker:
dependency: "direct main"
description:
Expand All @@ -94,6 +102,11 @@ packages:
description: flutter
source: sdk
version: "0.0.0"
flutter_driver:
dependency: transitive
description: flutter
source: sdk
version: "0.0.0"
flutter_lints:
dependency: "direct dev"
description:
Expand Down Expand Up @@ -136,6 +149,16 @@ packages:
url: "https://pub.dev"
source: hosted
version: "3.1.0"
fuchsia_remote_debug_protocol:
dependency: transitive
description: flutter
source: sdk
version: "0.0.0"
integration_test:
dependency: "direct dev"
description: flutter
source: sdk
version: "0.0.0"
json_annotation:
dependency: transitive
description:
Expand Down Expand Up @@ -287,6 +310,14 @@ packages:
url: "https://pub.dev"
source: hosted
version: "2.1.8"
process:
dependency: transitive
description:
name: process
sha256: c6248e4526673988586e8c00bb22a49210c258dc91df5227d5da9748ecf79744
url: "https://pub.dev"
source: hosted
version: "5.0.5"
rag_engine_flutter:
dependency: "direct overridden"
description:
Expand Down Expand Up @@ -331,6 +362,14 @@ packages:
url: "https://pub.dev"
source: hosted
version: "1.4.1"
sync_http:
dependency: transitive
description:
name: sync_http
sha256: "7f0cd72eca000d2e026bcd6f990b81d0ca06022ef4e32fb257b30d3d1014a961"
url: "https://pub.dev"
source: hosted
version: "0.3.1"
term_glyph:
dependency: transitive
description:
Expand Down Expand Up @@ -371,6 +410,14 @@ packages:
url: "https://pub.dev"
source: hosted
version: "1.1.1"
webdriver:
dependency: transitive
description:
name: webdriver
sha256: "2f3a14ca026957870cfd9c635b83507e0e51d8091568e90129fbf805aba7cade"
url: "https://pub.dev"
source: hosted
version: "3.1.0"
win32:
dependency: transitive
description:
Expand Down
2 changes: 2 additions & 0 deletions example/pubspec.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@ dev_dependencies:
flutter_test:
sdk: flutter
flutter_lints: ^5.0.0
integration_test:
sdk: flutter

dependency_overrides:
rag_engine_flutter:
Expand Down
Loading