ProEchoMem

Installation

Requirements

• Python 3.9+

pip install -r requirements.txt

Download Models

1. Whisper ASR Model:

# faster-distil-whisper-large-v3 will be downloaded automatically

2. Vision-Language Model (MiniCPM-V):

git lfs install
git clone https://huggingface.co/openbmb/MiniCPM-V-2_6-int4

3. ImageBind Model:

# Model weights will be downloaded automatically on first use

Quick Start

Environment Variables

Set the following environment variables:

# OpenAI API
export OPENAI_API_KEY="your-api-key"

# Optional: Azure OpenAI
export AZURE_OPENAI_API_KEY="your-azure-key"
export AZURE_OPENAI_ENDPOINT="your-endpoint"

Basic Usage

from ProEchoMem import ProEchoMem, QueryConfig

# Initialize system
pem = ProEchoMem(working_dir="./my_cache")
pem.load_caption_model()

# Stage 1: Construct episodic memory
pem.construct_episodic_memory(["video1.mp4", "video2.mp4"])

# Stage 2 & 3: Query with full pipeline
response = pem.query("What happens in the forest scene?")
print(response)

Custom Configuration

# Initialize with custom settings
pem = ProEchoMem(
    working_dir="./custom_cache",
    video_segment_length=60,  # Longer segments
    retrieval_topk_chunks=15,  # More retrieval
)

# Configure query
config = QueryConfig(
    mode="proechomem",
    top_k=15,
    enable_note_generation=True,
    note_generation_top_k=15,
)

response = pem.query("Describe the main character's journey.", config)

Benchmark Evaluation

python examples/benchmark_lvbench.py \
    --dataset path/to/lvbench.json \
    --video_dir path/to/videos \
    --output_dir ./results

Project Structure

ProEchoMem/
├── __init__.py                    # Main entry point
├── README.md                      # This file
├── requirements.txt               # Dependencies
├── LICENSE                        # MIT License
│
├── core/                          # Core system
│   ├── proechomem.py              # Main ProEchoMem class
│   ├── config.py                  # Configuration classes
│   └── types.py                   # Type definitions
│
├── memory/                        # Stage 1: Memory Construction
│   ├── construction.py            # Memory construction entry
│   ├── segmentation.py            # Video segmentation
│   ├── encoding/                  # Multimodal encoding
│   │   ├── asr.py                 # ASR transcription
│   │   ├── captioning.py          # Visual captioning
│   │   └── feature.py             # Feature encoding
│   ├── knowledge_extraction.py    # Entity/relation extraction
│   └── episodic_graph.py          # Graph construction
│
├── activation/                    # Stage 2: Memory Activation
│   ├── initial_trace.py           # Initial trace activation
│   ├── cognitive_probes.py        # 6 cognitive probes
│   └── parallel_activation.py     # Parallel retrieval
│
├── synthesis/                     # Stage 3: Echo Synthesis
│   ├── attentional_filter.py      # Attentional filtering
│   ├── structured_echo.py         # Structured echo synthesis
│   └── dual_context.py            # Response generation
│
├── retrieval/                     # Retrieval modules
│   ├── text_retrieval.py          # Text chunk retrieval
│   ├── entity_retrieval.py        # Entity retrieval
│   └── visual_retrieval.py        # Video segment retrieval
│
├── storage/                       # Storage backends
│   ├── kv_storage.py              # Key-value storage
│   ├── vector_storage.py          # Vector database
│   └── graph_storage.py           # Graph storage
│
├── llm/                           # LLM integration
│   └── openai_client.py           # OpenAI/Azure client
│
├── prompts/                       # Prompt templates
│   ├── extraction.py              # Knowledge extraction
│   ├── probe.py                   # Probe generation
│   ├── synthesis.py               # Echo synthesis
│   └── response.py                # Response generation
│
├── utils/                         # Utilities
│   ├── text.py                    # Text processing
│   ├── chunking.py                # Text chunking
│   └── async_utils.py             # Async utilities
│
└── examples/                      # Usage examples
    ├── basic_usage.py             # Basic usage demo
    └── benchmark_lvbench.py       # LVBench evaluation

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments

• GraphRAG for entity extraction prompts
• ImageBind for multimodal encoding
• MiniCPM-V for vision-language model
• faster-whisper for ASR