Feature image search (#5)

* Update our RAG to use CLIP embeddings to support the new multi-modl data.

* Working reverse image search using our new multimodal DB, with good results for the current prompt

* 1. Remove streaming endpoint
2. Actually pass images in to our vision model
3. Update code to use the latest 0.5.0 release of the vector db with separate image and text embeddings

Author: aniongithub

.env

@@ -1,4 +1,6 @@
 DEFAULT_MODEL="qwen3:0.6b"
+DEFAULT_IMAGE_MODEL="qwen2.5vl:3b"
+
 OLLAMA_URL="http://ollama:11434"
 DB_PATH="/data/enmemoryalpha_db"
 COLLECTION_NAME="memoryalpha"

.github/workflows/ci-build.yml

@@ -52,22 +52,6 @@ jobs:
           exit 1
         fi
         
-    - name: Test streaming endpoint
-      run: |
-        # Test the streaming endpoint
-        response=$(timeout 30 curl -s -N -H "Accept: text/event-stream" \
-          "http://localhost:8000/memoryalpha/rag/stream?question=What%20is%20a%20Transporter?&thinkingmode=DISABLED&max_tokens=50&top_k=3" \
-          | head -10)
-        
-        # Check if streaming response contains data events
-        if echo "$response" | grep -q "data:"; then
-          echo "✅ Streaming endpoint test passed"
-        else
-          echo "❌ Streaming endpoint test failed - no streaming data found"
-          echo "Response: $response"
-          exit 1
-        fi
-
     - name: Generate OpenAPI spec
       run: |
         # Download OpenAPI spec

Dockerfile

@@ -3,14 +3,16 @@ FROM python:3.12-slim-bullseye AS devcontainer
 ARG DEBIAN_FRONTEND=noninteractive
 
 RUN apt-get update &&\
-    apt-get -y install curl wget git
+    apt-get -y install jq curl wget git
 
 COPY ./requirements.txt /tmp/pip-tmp/
 RUN pip install --no-cache-dir -r /tmp/pip-tmp/requirements.txt \
     && rm -rf /tmp/pip-tmp
 
 WORKDIR /data
-RUN wget https://github.com/aniongithub/memoryalpha-vectordb/releases/latest/download/enmemoryalpha_db.tar.gz &&\
+
+ARG MEMORYALPHA_DB_RELEASE=v0.5.0
+RUN wget https://github.com/aniongithub/memoryalpha-vectordb/releases/download/${MEMORYALPHA_DB_RELEASE}/enmemoryalpha_db.tar.gz &&\
     tar -xzf enmemoryalpha_db.tar.gz &&\
     rm enmemoryalpha_db.tar.gz &&\
     chmod -R 0777 /data

README.md

@@ -6,7 +6,7 @@ A REST API for Retrieval-Augmented Generation (RAG) over Star Trek's MemoryAlpha
 
 ## Overview
 
-This project provides a streaming REST API that enables natural language queries over the comprehensive Star Trek MemoryAlpha database. It uses the vectorized database from [memoryalpha-vectordb](https://github.com/aniongithub/memoryalpha-vectordb) and combines it with local LLMs via Ollama to provide accurate, context-aware responses about Star Trek lore.
+This project provides a REST API that enables natural language queries over the comprehensive Star Trek MemoryAlpha database. It uses the vectorized database from [memoryalpha-vectordb](https://github.com/aniongithub/memoryalpha-vectordb) and combines it with local LLMs via Ollama to provide accurate, context-aware responses about Star Trek lore.
 
 The system implements:
 
@@ -169,7 +169,7 @@ graph TD
 
 ### Components
 
-- **FastAPI**: REST API framework with streaming support
+- **FastAPI**: REST API framework and OpenAPI spec generation
 - **ChromaDB**: Vector database for document storage and retrieval
 - **Ollama**: Local LLM inference server
 - **Cross-Encoder**: Document reranking for improved relevance

api/main.py

@@ -2,8 +2,8 @@
 from contextlib import asynccontextmanager
 from fastapi import FastAPI
 from .memoryalpha.health import router as health_router
-from .memoryalpha.stream import router as stream_router
 from .memoryalpha.ask import router as ask_router
+from .memoryalpha.identify import router as identify_router
 
 # Configure logging
 logging.basicConfig(level=logging.INFO)
@@ -21,5 +21,5 @@ async def lifespan(app: FastAPI):
 app = FastAPI(lifespan=lifespan)
 
 app.include_router(health_router)
-app.include_router(stream_router)
-app.include_router(ask_router)
+app.include_router(ask_router)
+app.include_router(identify_router)

api/memoryalpha/identify.py

@@ -0,0 +1,31 @@
+from fastapi import APIRouter, File, UploadFile, Query
+from fastapi.responses import JSONResponse
+import tempfile
+import os
+from .rag import MemoryAlphaRAG
+
+router = APIRouter()
+
+# Singleton or global instance for demo; in production, manage lifecycle properly
+rag_instance = MemoryAlphaRAG()
+
+@router.post("/memoryalpha/rag/identify", summary="Multimodal Image Search")
+def identify_endpoint(
+    file: UploadFile = File(...),
+    top_k: int = Query(5, description="Number of results to return")
+):
+    """
+    Accepts an image file upload, performs multimodal image search, and returns results.
+    """
+    try:
+        # Save uploaded file to a temp location
+        with tempfile.NamedTemporaryFile(delete=False, suffix=os.path.splitext(file.filename)[-1]) as tmp:
+            tmp.write(file.file.read())
+            image_path = tmp.name
+        # Perform image search
+        results = rag_instance.search_image(image_path, top_k=top_k)
+        # Clean up temp file
+        os.remove(image_path)
+        return JSONResponse(content=results)
+    except Exception as e:
+        return JSONResponse(status_code=500, content={"error": str(e)})