DSPy-Multi-Document-Agents

A multi-agent document processing system built with DSPy. Partitions documents into per-document agents that independently evaluate and answer queries, coordinated by a master agent with query planning and reranking.

Architecture

User query
  -> QueryPlanner (selects relevant document agents)
    -> DocumentAgents (evaluate relevance, extract answers in parallel)
      -> RerankModule (reorders results by score)
        -> MasterAgent (returns top answer with citations)

Components

Component	Description
MasterAgent	Orchestrates query flow across document agents
QueryPlanner	DSPy ChainOfThought module that selects which agents to involve
DocumentAgent	Per-document agent that evaluates queries, extracts answers, and generates responses via Claude
RerankModule	Reranks initial retrieval scores using Qdrant context
RerankingOptimizer	Uses DSPy's BootstrapFewShotWithRandomSearch to optimize reranking

Dependencies

Dependency	Purpose
DSPy	LLM orchestration and prompt optimization
Qdrant	Vector database for document embeddings
LlamaIndex	Document loading and vector store indexing
sentence-transformers (all-MiniLM-L6-v2)	Query encoding
Anthropic Claude (claude-3-haiku)	LLM for query planning, evaluation, answer generation
unstructured	Document parsing

Requirements

• Python >= 3.8
• Running Qdrant instance (default: localhost:6333)
• ANTHROPIC_API_KEY environment variable

Setup

git clone https://github.com/jmanhype/DSPy-Multi-Document-Agents.git
cd DSPy-Multi-Document-Agents
pip install -r requirements.txt
# Start Qdrant (e.g., via Docker)
docker run -p 6333:6333 qdrant/qdrant
export ANTHROPIC_API_KEY="your-key"
python main.py

By default, main.py loads documents from docs/latest.md (configurable via DOCUMENT_PATH env var).

How queries are processed

1. Documents are loaded, partitioned, embedded, and stored in Qdrant
2. A DocumentAgent is created for each document partition
3. QueryPlanner uses ChainOfThought to select relevant agents
4. Selected agents evaluate the query against their content
5. RerankModule adjusts scores using retrieved context
6. MasterAgent picks the top-scoring agent and asks it to generate a full answer with citations

Limitations

• The reranking logic is simplistic (adds context length to initial score)
• Relevance scoring uses keyword heuristics ("not relevant" / "partially relevant" string matching), not a trained classifier
• Query decomposition splits on "and"/"or" keywords only
• Similarity search uses Jaccard similarity, not semantic embeddings
• No web UI; runs as a CLI script
• The Nextra documentation site (pages/) describes the architecture but the docs app requires separate npm install && npm run dev

Documentation site

The pages/ directory contains a Nextra (Next.js) documentation site:

npm install
npm run dev
# Open http://localhost:3000

License

MIT