feat(release): 0.18.6 final polishing

.pubignore

@@ -54,6 +54,8 @@ refrag-develop.md
 # 테스트 드라이버
 test_driver/
 integration_test/
+example/test_driver/
+example/integration_test/
 
 # 플랫폼별 디렉토리 (rust_builder에서 제공, 루트 레벨만 제외)
 /android/
@@ -70,3 +72,5 @@ example/assets/evalsets/
 
 # Example evaluation/performance runners (local only)
 example/lib/*_runner.dart
+example/lib/profiling/
+example/test/profiling/

CHANGELOG.md

@@ -5,6 +5,20 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## 0.18.6
+* **PDF extraction UX**:
+  - Added OCR-needed classification helpers for scanned/image-only PDF extraction errors.
+  - Keeps mixed scanned + partially failed PDFs on the OCR-recoverable path while leaving fully failed/corrupt PDFs as raw extraction errors.
+  - Clarified production support boundaries: text-layer PDFs are production-ready, scanned PDFs require app-provided OCR, and DOCX remains beta.
+* **Release-path validation**:
+  - Added i8 hot-path recall/fidelity safety nets for the shipped `vector_quant_i8` path.
+  - Runs the shipped `vector_faer,vector_quant_i8` native feature combo in CI with fail-closed checks for renamed or skipped safety nets.
+  - Logs corrupt f32 embedding blobs with row ids during HNSW rebuild instead of silently dropping them.
+* **Packaging**:
+  - Excludes profiler-only example harnesses and tests from the published archive.
+* **Compatibility**:
+  - Bumped dependency constraint to `rag_engine_flutter: ^0.18.4` so consumers receive the matching native hardening patch.
+
 ## 0.18.5
 * **PDF extraction quality**:
   - Bumped dependency constraint to `rag_engine_flutter: ^0.18.3` so consumers receive the native PDF extraction improvements.

README.md

@@ -50,14 +50,16 @@ Data never leaves the user's device. Perfect for privacy-focused apps (journals,
 
 | Category | Features |
 |:---------|:---------|
-| **Document Input** | PDF, DOCX, Markdown, Plain Text with smart dehyphenation; file-path and UTF-8 ingest fast paths |
+| **Document Input** | Text-layer PDF, Markdown, Plain Text, and beta DOCX support; file-path and UTF-8 ingest fast paths |
 | **Chunking** | Plain-text paragraph/line chunking with heading-aware split and tokenizer hard guard; Markdown structure-aware chunking with header-path metadata |
 | **Search** | HNSW vector + BM25 keyword hybrid search with RRF fusion; metadata-first search with explicit context/chunk hydration |
 | **Storage** | SQLite persistence, HNSW Index persistence (fast startup), connection pooling, resumable indexing |
 | **Collections** | Collection-scoped ingest/search/rebuild via `inCollection('id')` |
 | **Performance** | Rust core, 10x faster tokenization, thread control, memory optimized |
 | **Context** | Engine-tokenizer exact context budget, adjacent chunk expansion, single source mode |
 
+**Support boundaries:** text-layer PDFs are production-ready. Scanned or image-only PDFs should be routed through an OCR layer before indexing. DOCX extraction is available for early adopters, but complex DOCX layouts such as tables, headers, and footnotes should be treated as beta.
+
 ---
 
 ## Requirements
@@ -179,8 +181,17 @@ try {
       lower.endsWith('.markdown')) {
     await MobileRag.instance.addDocumentUtf8(bytes, name: fileName);
   } else {
-    final text = await DocumentParser.parse(bytes);
-    await MobileRag.instance.addDocument(text, name: fileName);
+    try {
+      final text = await DocumentParser.parse(bytes);
+      await MobileRag.instance.addDocument(text, name: fileName);
+    } catch (error) {
+      if (DocumentParser.isOcrRequiredPdfExtractionError(error)) {
+        throw UnsupportedError(
+          DocumentParser.userMessageForExtractionError(error),
+        );
+      }
+      rethrow;
+    }
   }
 }
 ```

docs/guides/faq.md

@@ -109,7 +109,7 @@ await MobileRag.instance.rebuildIndex();
 
 ### Q: Memory usage is high
 
-- **Prefer file-path ingest for large local files**: Use `addDocumentFromFile(path)` for PDF, DOCX, Markdown, and text files when you have a stable path.
+- **Prefer file-path ingest for large local files**: Use `addDocumentFromFile(path)` for text-layer PDFs, Markdown, and text files when you have a stable path. DOCX is beta, and scanned/image-only PDFs need OCR before indexing.
 - **Use UTF-8 ingest when bytes are already loaded**: Use `addDocumentUtf8(bytes)` for text/Markdown bytes instead of converting them to a Dart `String` first.
 - **Limit document count**: 10K+ may cause degradation.
 - **Use INT8 models**: Smaller models reduce memory pressure.

docs/guides/quick_start.md

@@ -111,13 +111,17 @@ await MobileRag.instance.addDocument(
   'Flutter is Google\'s UI toolkit for building beautiful apps.',
 );
 
-// Add PDF/DOCX or other local files by path.
+// Add text-layer PDFs, beta DOCX files, or other local files by path.
 // This lets Rust read and chunk the file directly.
 await MobileRag.instance.addDocumentFromFile(
   'document.pdf',
   name: 'document.pdf',
 );
 
+// Scanned/image-only PDFs require OCR before indexing. If extraction fails,
+// use DocumentParser.isOcrRequiredPdfExtractionError(error) to route the file
+// to your app's OCR flow. DOCX extraction is beta for complex layouts.
+
 // If a stable local path is not available, fall back to bytes/text ingestion.
 final bytes = await File('notes.md').readAsBytes();
 await MobileRag.instance.addDocumentUtf8(bytes, name: 'notes.md');

docs/guides/troubleshooting.md

@@ -154,6 +154,29 @@ await MobileRag.instance.rebuildIndex();
 
 ---
 
+### "PDF requires OCR"
+
+**Symptom:** PDF ingestion fails with a message that the file may be scanned or image-based.
+
+**Cause:** The PDF does not contain enough extractable text for semantic indexing. Mobile RAG Engine does not run OCR internally.
+
+**Solution:** Route the file through your app's OCR flow, then index the recognized text:
+```dart
+try {
+  await MobileRag.instance.addDocumentFromFile(file.path, name: fileName);
+} catch (error) {
+  if (DocumentParser.isOcrRequiredPdfExtractionError(error)) {
+    // Replace this with your app-specific OCR implementation.
+    final recognizedText = await runYourAppOcrPipeline(file);
+    await MobileRag.instance.addDocument(recognizedText, name: fileName);
+  } else {
+    rethrow;
+  }
+}
+```
+
+---
+
 ### "Out of memory" on large documents
 
 **Symptom:** App crashes when processing large PDFs

docs/perf/ondevice-query-profiler/PLAN.md

@@ -20,7 +20,7 @@
 | P2 | example integration_test wiring + A/B fixture builder | LOC-67 | device |
 | P3 | Segment timing loop + 3 scenarios + query_metrics snapshot | LOC-68 | device |
 | P4 | JSON/CSV export to app-docs + structured log + run metadata | LOC-69 | device + host(serialize) |
-| P5 | (conditional) Phase-2 drill-down per dominant bucket | LOC-70 | TBD by P3/P4 data |
+| P5 | Phase-2 quality/latency drill-down | LOC-70 | P5-① recall complete; P5-②~④ deferred after 0.18.6 |
 
 Each PR: branch from main, commit (no Claude attribution), open PR, CI green (`cargo test -- --test-threads=1` unaffected — Dart-only), **user merges**. Fill `PRn.md` + README row before merge.
 

docs/perf/ondevice-query-profiler/PR-P2.md

@@ -2,7 +2,7 @@
 
 - 브랜치: `feat/loc-67-profiler-fixture`
 - Linear: [LOC-67](https://linear.app/loceract/issue/LOC-67)
-- 상태: 🟦 진행 (PR 열림, 기기 스모크 green)
+- 상태: 🟩 머지(#71, 기기 스모크 green)
 
 ## 스코프
 실 ONNX 자산이 있는 example 앱에서 구동하는 디바이스 프로파일러의 기반.

docs/perf/ondevice-query-profiler/PR-P3.md

@@ -2,7 +2,7 @@
 
 - 브랜치: `feat/loc-68-profiler-timing`
 - Linear: [LOC-68](https://linear.app/loceract/issue/LOC-68)
-- 상태: 🟦 진행 (PR 열림 예정, iPhone 실기 profile 런 green)
+- 상태: 🟩 머지(#72, iPhone 실기 profile 런 green)
 
 ## 스코프
 실 ONNX 자산 example 앱에서 쿼리 단계를 **격리 계측**하는 디바이스 프로파일러 본체.

docs/perf/ondevice-query-profiler/PR-P4.md

@@ -2,7 +2,7 @@
 
 - 브랜치: `feat/loc-69-profiler-export` (P3 위에 스택)
 - Linear: [LOC-69](https://linear.app/loceract/issue/LOC-69)
-- 상태: 🟦 진행 (PR 열림 예정, iPhone 실기 profile 런 green) — **baseline 산출 = 1차 목표 달성**
+- 상태: 🟩 머지(#75 rescue, iPhone 실기 profile 런 green) — **baseline 산출 = 1차 목표 달성**
 
 ## 스코프
 - `example/lib/profiling/profile_export.dart` — 리포트를 앱 documents dir에 `query_profile_<ts>.json/.csv`로 flush 기록 + 실행당 `PROFILE` 로그 1줄 + dir/파일명 로그(추출용). 물리기기 추출은 Xcode Download Container 또는 `xcrun devicectl`(simctl은 시뮬전용).
@@ -26,7 +26,7 @@ I/O(query_metrics): full_hydrate_rows pure_warm 300 / filtered 90 / pure_cold 10
 ## ⚠️ 측정 범위 (헤드라인 정정 — latency ≠ quality)
 - 본 baseline은 **지연(latency) 분해**만 측정한다. **검색 품질(recall)은 측정하지 않는다.**
 - LOC-64의 `recall@10 = 0.997`은 **i8 exact-scan vs f32 전수조사의 수학적 충실도**(양자화 커널 정확도)일 뿐, **실제 출시 경로의 HNSW 그래프 e2e 하이브리드 리콜이 아니다** — HNSW는 `dequantize_i8_to_f32`로 복원한 찌그러진 벡터 공간 위에서 그래프를 탐색하므로 별개. 즉 "i8가 빠르다 + 0.997"을 "출시 검색 품질이 우수하다"로 읽으면 안 된다.
-- **실제 프로덕션 e2e 하이브리드 리콜은 미측정** → **P5(LOC-70)의 1순위 타깃**(폰에서 `[f32 순수 원본 전수조사 top-K]` 대비 `[i8-HNSW + BM25 RRF top-K]` 교집합률 산출). 90% 미만이면 모바일용 `M`/`ef_search` 상향 튜닝 필요.
+- **후속 업데이트:** P5-①에서 실제 프로덕션 e2e 리콜을 측정했고, vector-only recall@10=1.00으로 HNSW/i8 품질 게이트를 통과했다. hybrid recall@10=0.08은 순수-vector GT 대비 BM25/RRF reorder 진단으로 해석한다.
 
 ## 받은 피드백 / 한계
 - 어드버서리얼 리뷰 HIGH: 초기 추출 안내가 simctl(시뮬전용)이었음 → 물리기기는 Xcode Download Container / `devicectl`로 정정.
@@ -35,5 +35,4 @@ I/O(query_metrics): full_hydrate_rows pure_warm 300 / filtered 90 / pure_cold 10
 
 ## 리스크 / 롤백 / 다음
 - 동작 코드 변경 없음(example 프로파일링 export). 롤백: PR revert.
-- ⚠️ **스택 PR**: P3(#LOC-68) 위에 스택. P3 먼저 머지 후 본 PR을 main으로 retarget(아니면 orphan).
-- 다음: **P5(LOC-70)** Phase-2 — warm은 embed(ONNX) 지배라 Rust 벡터 작업 불요; cold가 중요하면 activate(bm25_rebuild vs hnsw_load) 분해. 데이터 게이트 충족.
+- P5-① 품질 게이트 완료 후 남은 P5-②~④는 0.18.6 이후의 측정 심화 작업으로 보류.

docs/perf/ondevice-query-profiler/PR-P5-1.html

@@ -291,8 +291,8 @@ <h2>Recommended Next Steps</h2>
       <tbody>
         <tr>
           <td>Next</td>
-          <td>Proceed to P5-2 activate breakdown.</td>
-          <td>The vector-only quality gate passed, while cold activate remains the latency gate.</td>
+          <td>Defer P5-2 activate breakdown until after 0.18.6.</td>
+          <td>The vector-only quality gate passed; cold activate remains the next latency gate, but not a release blocker.</td>
         </tr>
         <tr>
           <td>Later</td>

docs/perf/ondevice-query-profiler/README.md

@@ -27,7 +27,7 @@ vector_math 커널 슬라이스는 거의 최적임을 확인했으나 **온디
 | P3 | 세그먼트 타이밍 + 3시나리오 + metrics 스냅샷 | [LOC-68](https://linear.app/loceract/issue/LOC-68) | 🟩 머지(#72, [PR-P3.md](PR-P3.md), 기기 green) |
 | P4 | JSON/CSV export + 로그 + 메타 (baseline 산출) | [LOC-69](https://linear.app/loceract/issue/LOC-69) | 🟩 머지(#75 rescue, [PR-P4.md](PR-P4.md), 기기 green) |
 | P5-① | e2e hybrid recall@10 — 품질 | [LOC-70](https://linear.app/loceract/issue/LOC-70) | 🟩 완료([PR-P5-1.html](PR-P5-1.html), vector-only=1.00 / hybrid=0.08) |
-| P5-②~④ | activate 분해 / 동시성 / SQLite scale | [LOC-70](https://linear.app/loceract/issue/LOC-70) | ⏭ 다음 순서 |
+| P5-②~④ | activate 분해 / 동시성 / SQLite scale | [LOC-70](https://linear.app/loceract/issue/LOC-70) | ⏸ 0.18.6 이후 측정 심화 |
 
 ## 규약 (프로젝트 공통)
 - CI: `cargo test -- --test-threads=1`. 커밋/PR에 Claude 귀속 미포함. PR은 열고 CI green까지만, 머지는 본인.

docs/perf/vector-math-refactor/PR1.md

@@ -2,7 +2,7 @@
 
 - 브랜치: `feat/loc-59-vector-math-bench`
 - Linear: [LOC-59](https://linear.app/loceract/issue/LOC-59)
-- 상태: 🟦 진행 (PR 열림, CI green 대기)
+- 상태: 🟩 머지(#64)
 
 ## 스코프 (무엇을/왜)
 faer 삭제 **전에** 성능 베이스라인을 캡처하고, fused가 faer와 등가임을 박아 PR2를 안전화하기 위함.
@@ -42,5 +42,5 @@ faer 삭제 **전에** 성능 베이스라인을 캡처하고, fused가 faer와
 - 벤치/패리티는 CI에서 안 돎(`bench`·`vector_faer` 미설정) → 로컬 실행/기록이 본 PR의 산출.
 
 ## 결정 로그
-- **PR2 전제 반증** → PR2를 “faer 제거”에서 **“faer 유지 + N2(CI 커버리지) [+ 선택 N1 무할당화]”로 피벗** 후보. 사용자 결정 대기.
+- **PR2 전제 반증** → PR2를 “faer 제거”에서 **“faer 유지 + N2(CI 커버리지)”로 피벗**했고, PR2(#65)에서 완료.
 - 디바이스 캐비엇: 위 수치는 개발기(Apple Silicon). 방향(faer 우위)은 스칼라-vs-SIMD 차이라 폰 NEON에서도 견고할 것으로 예상하나, 크기는 온디바이스 프로파일로 확인 권장.

docs/perf/vector-math-refactor/PR2.md

@@ -2,7 +2,7 @@
 
 - 브랜치: `feat/loc-60-faer-ci-coverage`
 - Linear: [LOC-60](https://linear.app/loceract/issue/LOC-60)
-- 상태: 🟦 진행 (PR 열림, CI green 대기)
+- 상태: 🟩 머지(#65)
 
 ## 재설계 배경
 PR1([PR1.md](PR1.md)) 벤치가 원안("faer 제거 → fused 통일")의 전제를 반증 — faer가 2–8× 빠름.
@@ -22,10 +22,10 @@ PR1([PR1.md](PR1.md)) 벤치가 원안("faer 제거 → fused 통일")의 전제
 - N2 갭: **닫힘** — 출시 faer+quant 경로가 CI에서 빌드 + 테스트됨.
 - 로컬 검증: `bash -n` OK; faer 테스트 4개(패리티 포함) green, fail-closed PASS; `cargo build --release
   --features vector_faer,vector_quant_i8` 성공(50s, 기존 dead_code 경고만).
-- CI: (PR #__ green 후 갱신)
+- CI: PR #65 green 후 머지.
 
 ## 받은 피드백 (리뷰)
-- (PR 리뷰 후 갱신)
+- faer 제거 대신 출시 faer 백엔드를 CI에서 검증하는 방향으로 피벗했고, PR1 측정 결과와 일치.
 
 ## 리스크 / 롤백
 - CI native 잡 시간 증가(faer 2회 컴파일: test+release 프로파일). 게이트 가치 대비 수용.

docs/perf/vector-math-refactor/PR6-plan-i8-measure-parity-net.md

@@ -550,27 +550,27 @@ git commit -m "ci(vector_quant): run i8 ε/recall/fidelity nets on shipped faer+
 
 - [ ] **Step 1: Create `PR6.md` with the measured results**
 
-Create `docs/perf/vector-math-refactor/PR6.md` (fill `<...>` from Task 2 Step 5 and Task 3 Step 4):
+Create `docs/perf/vector-math-refactor/PR6.md` with the measured values from Task 2 Step 5 and Task 3 Step 4:
 
 ```markdown
 # PR6 — i8 출시 핫패스 측정 + ε/recall/fidelity 안전망 (N: 측정 먼저)
 
 - 브랜치: `feat/loc-64-i8-measure-parity-net`
 - Linear: [LOC-64](https://linear.app/loceract/issue/LOC-64)
-- 상태: 🟦 진행 (PR 열림, CI green 대기)
+- 상태: 🟩 머지(#68)
 - 설계: [PR6-spec-i8-measure-parity-net.md](PR6-spec-i8-measure-parity-net.md)
 
 ## 스코프 (비파괴 — 커널/양자화 0줄 변경)
 출시 핫패스(i8 `cosine_with_query_norm_i8_blob`)에 PR1 패턴 적용: 측정 + 수치 ε 네트 + recall@k floor + 코사인 fidelity 네트 + CI fail-closed.
 
-## 결과 (측정)
-- **i8 핫커널 마이크로벤치** (dim별, ns): 384=<...> / 768=<...> / 1024=<...> / 1536=<...>
-- **스캔(2000×768) 비교**: `exact_scan[faer]`(f32 decode+cosine)=<...> µs vs `exact_scan_i8`(i8 blob)=<...> µs → i8가 f32-faer 대비 **<...>×**.
+## 결과 (측정, dev arm64)
+- **i8 핫커널 마이크로벤치** (ns): 384=7.87 / 768=14.97 / 1024=21.12 / 1536=31.28
+- **스캔(2000×768) 비교**: `exact_scan[faer]`(f32 decode+cosine)=452.82 µs vs `exact_scan_i8`(i8 blob)=29.98 µs → i8가 f32-faer 대비 **≈15.1×**.
 - **수치 ε 네트**: 차원 {1,2,3,16,384,768,1024,1536}에서 kernel ≈ f64 참조, ε=1e-4 green.
 - **핵심 발견**: i8 per-vector 양자화는 768d에서 **recall@10 ≈ 0.997**(=319/320, 거의 무손실) — '민감 밴드'는 출시 설정에서 도달 불가이며 강제 시 비대표적. 따라서 게이트는 이 높은 baseline을 잠금.
 - **recall@k floor 네트**: N=2000, Q=32, dim=768, k=10, clusters=16 → 측정 recall@10 = **0.996875**(dev arm64, CI 확인), FLOOR = **0.98** (= floor(X−0.02)). GT는 f64(플랫폼 jitter 제거), 전순서 `(score desc, index asc)`.
 - **코사인 fidelity 네트**: `max|cosine_i8 − cosine_f32_true|` = **0.00121**, 게이트 = **0.005** (≈4× baseline). 완전 결정론적·민감.
-- **CI**: `--features "vector_quant_i8,vector_faer" -- --test-threads=1` fail-closed + 3개 네트 이름별 가드 (N=<...> passed).
+- **CI**: `--features "vector_quant_i8,vector_faer" -- --test-threads=1` fail-closed + 3개 네트 이름별 가드, 7 passed.
 
 ## 받은 피드백 (리뷰 / 사전검증)
 - 설계 리뷰: N=2000/k=10, 전순서 타이브레이크, 출시 트리(faer+quant) CI.
@@ -591,7 +591,7 @@ Create `docs/perf/vector-math-refactor/PR6.md` (fill `<...>` from Task 2 Step 5
 In `docs/perf/vector-math-refactor/README.md`, add after the PR5 row line:
 
 ```markdown
-| PR6 | i8 출시 핫패스 **측정 + ε/recall/fidelity 안전망** | i8 검증갭 | 낮음(비파괴) | main(#67) | [LOC-64](https://linear.app/loceract/issue/LOC-64) | 🟦 진행([PR6.md](PR6.md)) |
+| PR6 | i8 출시 핫패스 **측정 + ε/recall/fidelity 안전망** | i8 검증갭 | 낮음(비파괴) | main(#67) | [LOC-64](https://linear.app/loceract/issue/LOC-64) | 🟩 머지(#68, [PR6.md](PR6.md)) |
 ```
 
 - [ ] **Step 3: Commit**
@@ -649,6 +649,6 @@ Expected: all checks pass. **Do NOT merge** — report "PR opened, CI green" and
 ## Self-Review (filled by plan author)
 
 - **Spec coverage**: §3 bench → Task 3; §4 ε net → Task 1; §5 quality net → Task 2 (recall floor + fidelity, per the approved Net-2 redesign); §6 CI → Task 4; §7 tracking → Task 5 + issue created; §8 acceptance → Tasks 1–6; non-goal (0 kernel changes) → only `mod tests`, `bench_api`, `benches`, CI, docs touched. ✅ (Spec §5/§8/§9/§10 updated to the recall-floor + fidelity design.)
-- **Placeholders**: `MIN_RECALL`/`MAX_COS_ERR`/bench numbers are *measure-first outputs* with exact derivation + compile-time `const _` guards (Task 2 Steps 4–5), not vague TODOs. PR6.md `<...>` are explicitly "fill from measured results." ✅
+- **Placeholders**: `MIN_RECALL`/`MAX_COS_ERR`/bench numbers are *measure-first outputs* with exact derivation + compile-time `const _` guards (Task 2 Steps 4–5), not vague TODOs. The final PR6 journal uses measured values, not placeholders. ✅
 - **Type consistency**: `order_desc<T: PartialOrd>` used on both `(usize,f64)` (GT) and `(usize,f32)` (i8); `cosine_f64_true`/`ref_cosine_i8_f64`/`clustered_corpus`/`quantize_f32_to_i8`/`l2_norm_i8`/`i8_blob_from_slice`/`cosine_with_query_norm_i8_blob` match real `vector_quant.rs` signatures (verified). bench_api wrappers match. ✅
 - **Verification-driven fixes applied**: f64 GT (jitter), recall-floor not forced-band (saturation), const guards + CI name guards (vacuous gate), `pub(crate) mod` wording, flat `X−0.02` floor. ✅

docs/perf/vector-math-refactor/PR6.md

@@ -2,7 +2,7 @@
 
 - 브랜치: `feat/loc-64-i8-measure-parity-net`
 - Linear: [LOC-64](https://linear.app/loceract/issue/LOC-64)
-- 상태: 🟦 진행 (PR 열림, CI green 대기)
+- 상태: 🟩 머지(#68)
 - 설계: [PR6-spec-i8-measure-parity-net.md](PR6-spec-i8-measure-parity-net.md) · 계획: [PR6-plan-i8-measure-parity-net.md](PR6-plan-i8-measure-parity-net.md)
 
 ## 스코프 (비파괴 — 커널/양자화 0줄 변경)