Predefined Semantic Vocabulary layer with Cached AI Tags

[rohan-pandeyy]

Closes #1388

Summary

Adds a curated, precomputed vocabulary of semantic labels (beach, diwali, sunset, …) on top of the existing SigLIP2 embedding pipeline. Photos get searchable, browsable AI tags beyond YOLO's 80 classes, with common-word searches served from an instant SQL cache instead of a live matmul.

Design

Semantic labels reuse the existing YOLO tag flow instead of a parallel path:

• Labels register as mappings rows at class_id >= 1000 (YOLO keeps 0–79); scoring writes plain image_classes rows with a new nullable score column. Tag search, chips, and person-view tag lists work with zero consumer changes.
• semantic_labels is a definition + cache table: per-label category, 2–4 caption-shaped descriptions (source of truth), and a cached label embedding (renormalized mean of description embeddings — attacks SigLIP2's bare-noun weakness text-side).
• Scoring: one matmul per image against the label matrix → bucket threshold → top-15 stored with scores. Invalidation via a content signature (checkpoint + vocab + thresholds + K); any change re-scores wholesale, which is cheaper than delta tracking.
• Display cut: an image_classes_display view caps chips at the top-5 semantic tags per image; search matching keeps the full stored set.
• All passes are self-gating and idempotent, triggered at startup, folder sync, and semantic model install.

Vocabulary

395 labels (194 scene / 139 object / 45 event / 17 attribute), derived from Places365 + Open Images, hand-pruned (COCO-80/person exclusion, synonym collapse, junk removal), plus hand-written events (incl. diwali, holi, ganesh chaturthi, durga puja, navratri, mehndi, eid) and attributes (sunset, night, selfie, …). 1,085 authored descriptions.

Calibration (measured, not guessed)

151-image stratified eval set (Wikimedia Commons, attribution recorded), scored through the production preprocessing pipeline:

bucket	top-1	top-5	threshold
scene	0.52	0.83	5e-5
object	0.85	0.98	1e-4
event	0.64	0.88	5e-5
attribute	0.67	0.96	1e-4

Ensembled label scores sit ~2 orders of magnitude below the live-query path, so these floors are deliberately not SIGLIP2_MATCH_THRESHOLD. Null-baseline contrast was evaluated and rejected (never beat absolute thresholds). Full report: backend/scripts/vocabulary/calibration_report.json.

Migration

The old semantic_labels/image_semantic_labels shells shipped with no writers (guaranteed empty) → detected and drop-recreated. Populated tables get guarded ALTERs (image_classes.score, image_embeddings.scored_signature).

Testing

• 155 backend tests passing; 18 new (migration, upsert idempotency, embedding-cache staleness, scoring/top-K/signature invalidation, display cut).
• Verified end-to-end with real models: eval images produce correct tags (beach__1 → coast, beach, island, ocean) and search?tag=beach / balloon / case-insensitive queries hit the cache.

Out of scope

EXIF write-back (later PR, reads the same rows with a stricter cut); score-ordered tag-search results.

AI Usage Disclosure:

We encourage contributors to use AI tools responsibly when creating Pull Requests. While AI can be a valuable aid, it is essential to ensure that your contributions meet the task requirements, build successfully, include relevant tests, and pass all linters. Submissions that do not meet these standards may be closed without warning to maintain the quality and integrity of the project. Please take the time to understand the changes you are proposing and their impact. AI slop is strongly discouraged and may lead to banning and blocking. Do not spam our repos with AI slop.

Check one of the checkboxes below:

• This PR does not contain AI-generated code at all.
• This PR contains AI-generated code. I have read the AI Usage Policy and this PR complies with this policy. I have tested the code locally and I am responsible for it.

I have used the following AI models and tools: Claude

Checklist

• My PR addresses a single issue, fixes a single bug or makes a single improvement.
• My code follows the project's code style and conventions
• If applicable, I have made corresponding changes or additions to the documentation
• If applicable, I have made corresponding changes or additions to tests
• My changes generate no new warnings or errors
• I have joined the Discord server and I will share a link to this PR with the project maintainers there
• I have read the Contribution Guidelines
• Once I submit my PR, CodeRabbit AI will automatically review it and I will address CodeRabbit's comments.
• I have filled this PR template completely and carefully, and I understand that my PR may be closed without review otherwise.

Summary by CodeRabbit

• New Features
  • Added curated semantic vocabulary with semantic image tagging/scoring and configurable top‑K display.
  • Semantic vocabulary sync, embedding backfill, and image scoring now run automatically on startup and during folder processing.
• Bug Fixes
  • Improved consistency of displayed semantic tags across image views and tag-based queries.
  • Added safer incremental “needs scoring” detection to avoid unnecessary rescoring.
• Maintenance
  • Added tooling to build, calibrate, and reset semantic vocab/embeddings; expanded eval-set download support.
• Documentation & Tests
  • Updated semantic search and database schema docs; added comprehensive semantic integration tests.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: ce8d7c79-61e8-4ad9-b585-731617c79109

📥 Commits

Reviewing files that changed from the base of the PR and between aca1a15 and fe0e8c1.

📒 Files selected for processing (1)

• docs/backend/backend_python/semantic-search.md

🚧 Files skipped from review as they are similar to previous changes (1)

• docs/backend/backend_python/semantic-search.md

---

Walkthrough

Adds a curated 395-label semantic vocabulary, offline SIGLIP2 label and image scoring, cached embeddings and scoring signatures, database migration and display limits, calibration tooling, lifecycle integration, and tests.

Changes

Semantic vocabulary and pre-scoring

Layer / File(s)	Summary
Vocabulary contract and generation backend/app/config/settings.py, backend/app/data/semantic_vocabulary.json, backend/scripts/build_semantic_vocabulary.py, backend/scripts/vocabulary/raw_candidates.json	Defines semantic thresholds and top-K limits, ships the curated vocabulary, and generates candidate labels from Places365 and Open Images with YOLO exclusions.
Evaluation and calibration tooling backend/scripts/build_eval_set.py, backend/scripts/calibrate_semantic_vocabulary.py, backend/scripts/vocabulary/*, .gitignore	Downloads evaluation images, computes calibration metrics, writes a calibration report, and ignores local vocabulary artifacts.
Semantic schema and database storage backend/app/database/semantic_labels.py, backend/app/database/image_embeddings.py, backend/app/database/images.py, backend/app/database/faces.py	Migrates semantic label storage, caches embeddings, tracks scoring signatures, preserves YOLO rows, and limits displayed semantic tags.
Embedding and image scoring backend/app/utils/semantic_labels.py	Synchronizes vocabulary data, builds cached label embeddings, scores image embeddings in chunks, and persists semantic tags.
Lifecycle wiring and validation backend/main.py, backend/app/routes/*, backend/scripts/reset_embeddings.py, backend/tests/*	Runs semantic preprocessing during startup, model backfills, folder workflows, and resets, with coverage for migration, caching, scoring, display cuts, and ordering.
Documentation docs/backend/backend_python/*	Documents the active curated vocabulary, database schema, scoring invalidation, search behavior, and reset flow.

Estimated code review effort: 5 (Critical) | ~120 minutes

Sequence Diagram(s)

sequenceDiagram
  participant ApplicationStartup
  participant semantic_util_sync_vocabulary
  participant semantic_util_build_label_embeddings
  participant semantic_util_score_images
  participant SQLite
  ApplicationStartup->>semantic_util_sync_vocabulary: synchronize shipped vocabulary
  ApplicationStartup->>semantic_util_build_label_embeddings: build cached label embeddings
  semantic_util_build_label_embeddings->>SQLite: store label embeddings
  ApplicationStartup->>semantic_util_score_images: score embedded images
  semantic_util_score_images->>SQLite: write semantic scores and signatures

Loading

Possibly related PRs

• AOSSIE-Org/PictoPy#1377: Provides related SigLIP2 and embedding-table infrastructure extended here with pre-scoring.
• AOSSIE-Org/PictoPy#1382: Covers related semantic embedding workflow orchestration during model setup.

Suggested labels: Python, Documentation

Poem

A bunny found labels beneath the moon,
Scored little tags by afternoon.
Embeddings hopped through rows so neat,
Cached bright carrots made searches fleet.
“Top-K,” I whisper, “what a treat!”

🚥 Pre-merge checks | ✅ 3 | ❌ 1

❌ Failed checks (1 inconclusive)

Check name	Status	Explanation	Resolution
Linked Issues check	❓ Inconclusive	The core semantic-vocabulary, offline scoring, SQL-search, and calibration work matches #1388, but the summaries do not verify the required culturally diverse event labels.	Provide explicit evidence or tests that Diwali, Holi, Navratri, and Eid are included in the seeded vocabulary.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title is concise and matches the main change: adding a predefined semantic vocabulary layer with cached AI tags.
Out of Scope Changes check	✅ Passed	The added scripts, schema changes, tests, docs, and ignore rules all support the new semantic-vocabulary pipeline.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🧹 Nitpick comments (2)

backend/scripts/calibrate_semantic_vocabulary.py (1)

49-61: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Add type hints to build_label_matrix.

As per path instructions, ensure proper use of type hints. While this is an offline script, annotating top-level functions improves readability and maintainability.

💡 Proposed refactor to add type annotations

-def build_label_matrix(labels, text_model, tokenize):
+from typing import Any, Callable
+
+def build_label_matrix(
+    labels: list[dict],
+    text_model: Any,
+    tokenize: Callable[[str], tuple[np.ndarray, np.ndarray]],
+) -> np.ndarray:
     """Ensembled label vectors: renormalized mean of description embeddings."""

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@backend/scripts/calibrate_semantic_vocabulary.py` around lines 49 - 61, Add
type annotations to the build_label_matrix function, including parameter types
for labels, text_model, and tokenize, plus its NumPy array return type. Use
appropriate existing or standard typing symbols and preserve the current
embedding and normalization behavior.

Source: Path instructions

backend/app/utils/semantic_labels.py (1)

8-20: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Add type hints to _scoring_signature.

The public DB functions in this cohort are fully typed, but this helper's parameters are untyped. Adding hints keeps the module consistent and documents the meta/thresholds/label_matrix contract the signature hashing depends on.

As per path instructions ("Ensure proper use of type hints").

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@backend/app/utils/semantic_labels.py` around lines 8 - 20, Add complete type
annotations to the _scoring_signature helper, including model_version, top_k,
thresholds, meta, label_matrix, and its string return type. Use the existing
project types or appropriate concrete types to document the meta tuple
structure, threshold sequence, and NumPy label matrix contract without changing
the hashing behavior.

Source: Path instructions

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@backend/app/utils/semantic_labels.py`:
- Around line 8-20: Add complete type annotations to the _scoring_signature
helper, including model_version, top_k, thresholds, meta, label_matrix, and its
string return type. Use the existing project types or appropriate concrete types
to document the meta tuple structure, threshold sequence, and NumPy label matrix
contract without changing the hashing behavior.

In `@backend/scripts/calibrate_semantic_vocabulary.py`:
- Around line 49-61: Add type annotations to the build_label_matrix function,
including parameter types for labels, text_model, and tokenize, plus its NumPy
array return type. Use appropriate existing or standard typing symbols and
preserve the current embedding and normalization behavior.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: edf2b7e9-c44c-46bc-9390-c1344e9b5729

📥 Commits

Reviewing files that changed from the base of the PR and between 7af5eea and 5570d99.

📒 Files selected for processing (21)

• .gitignore
• backend/app/config/settings.py
• backend/app/data/semantic_vocabulary.json
• backend/app/database/faces.py
• backend/app/database/image_embeddings.py
• backend/app/database/images.py
• backend/app/database/semantic_labels.py
• backend/app/routes/folders.py
• backend/app/routes/models.py
• backend/app/utils/semantic_labels.py
• backend/main.py
• backend/scripts/build_eval_set.py
• backend/scripts/build_semantic_vocabulary.py
• backend/scripts/calibrate_semantic_vocabulary.py
• backend/scripts/reset_embeddings.py
• backend/scripts/vocabulary/calibration_report.json
• backend/scripts/vocabulary/eval_manifest.json
• backend/scripts/vocabulary/raw_candidates.json
• backend/tests/conftest.py
• backend/tests/test_models.py
• backend/tests/test_semantic_labels.py

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/backend/backend_python/semantic-search.md`:
- Line 83: Update the vocabulary-section link target in the semantic-search
documentation to use the renderer-generated fragment for the referenced heading,
or add an explicit matching anchor when necessary. Preserve the link text and
ensure the internal link resolves correctly.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 1008b000-eba8-4d3d-bede-72a0f49f225d

📥 Commits

Reviewing files that changed from the base of the PR and between 67bc643 and aca1a15.

📒 Files selected for processing (2)

• docs/backend/backend_python/database.md
• docs/backend/backend_python/semantic-search.md