WIP

Author: kwojcik-blockether

.gitignore

@@ -18,3 +18,4 @@ public
 .lsp
 *.log
 verification/**
+.clj-kondo

.serena/.gitignore

@@ -0,0 +1 @@
+/cache

.serena/memories/code_style_conventions.md

@@ -0,0 +1,62 @@
+# Code Style and Conventions
+
+## Core Development Rules
+
+### Code Quality Requirements
+- **Type hints required** for all code
+- **Public APIs must have docstrings**
+- **Functions must be focused and small**
+- **Follow existing patterns exactly**
+- **Line length: 120 chars maximum**
+- **Avoid `Any` type** - prefer typed classes inheriting from Pydantic `BaseModel`
+
+### File Organization
+- Every module should have files ending with `Core` and `internal` folder
+- **MAKE ALL properties in class private** by prepending `_` to variable name
+- **IF property should be public** then hide it using `_` and create a property function
+- **AVOID MAGIC NUMBERS** - instead create STATIC fields in class
+
+### Comment Style
+- **NO inline comments after code** - Never write comments on the same line as code (e.g., `x = 5  # this is bad`)
+- Comments should be on their own line above the code they describe
+- Docstrings are preferred over comments for function/class documentation
+
+### Class Design
+- **Classes with only static methods should not have instances created**
+  - Mark such classes appropriately (e.g., with abstract base class or clear documentation)
+  - Consider using module-level functions instead if appropriate
+
+### Forbidden Practices
+- Using `hasattr` and `getattr` by default
+- Use static types instead of dictionaries whenever possible and sensible
+
+### Testing Requirements  
+- **Framework**: pytest with anyio (preferred over asyncio)
+- **Test file naming**: Must end in `Test` postfix (e.g., `KnowledgeSearchCoreTest.py`)
+- **Test classes**: Always use classes in test files
+- **One test file per implementation file**
+
+### Test Quality Standards - NO WEAK TESTS!
+Tests MUST:
+- **Not have any `if` statements**
+- **Test real values and not only shape**
+- **Not use `try`/`catch` for testing** (prevents false positives)
+- **Have hardcoded values mostly** and not ranges like `len(expression) > magic_number`
+- **NO MAGIC NUMBERS** - put numbers in class constants
+
+### Exception Handling
+- **Always use `logger.exception()` instead of `logger.error()` when catching exceptions**
+- Don't include the exception in the message: `logger.exception("Failed")` not `logger.exception(f"Failed: {e}")`
+- **Catch specific exceptions** where possible:
+  - File ops: `except (OSError, PermissionError):`
+  - JSON: `except json.JSONDecodeError:`
+  - Network: `except (ConnectionError, TimeoutError):`
+- **Only catch `Exception` for**:
+  - Top-level handlers that must not crash
+  - Cleanup blocks (log at debug level)
+
+### Formatting Tools
+- **Ruff**: Primary linter and formatter
+- **Black**: Code formatting (line-length=120, target py312)
+- **isort**: Import sorting (black profile, line_length=120)
+- **MyPy**: Type checking with strict settings

.serena/memories/development_workflow.md

@@ -0,0 +1,82 @@
+# Development Workflow
+
+## Package Management - CRITICAL RULES
+
+### ✅ ALWAYS Use uv
+```bash
+uv add package                    # Add new dependency
+uv add --dev package             # Add dev dependency  
+uv run tool                      # Run any command
+uv run python3 script.py         # Run Python scripts
+```
+
+### ❌ FORBIDDEN Commands
+```bash
+uv pip install                   # NEVER use this
+pip install                      # NEVER use this
+uv add package@latest            # NEVER use @latest syntax
+```
+
+## Development Process
+
+### 1. Initial Setup
+```bash
+# Clone and install
+git clone <repo>
+uv sync                          # Install dependencies
+```
+
+### 2. Daily Development Cycle
+```bash
+# Before starting work
+poe format                       # Format code
+poe verify                       # Check types and linting
+
+# After making changes
+poe test                         # Run tests
+poe check                        # Full verification
+```
+
+### 3. Adding New Features
+1. **Check existing libraries first** - look at neighboring files, pyproject.toml
+2. **Follow existing patterns** - examine similar components
+3. **Use proper naming conventions** - `{Module}Core.py`, `{Module}Test.py`
+4. **Add comprehensive tests** - one test file per implementation file
+
+### 4. Error Resolution Order
+1. **Format first**: `poe format`
+2. **Fix type errors**: Add type hints, None checks
+3. **Fix linting**: Remove unused imports, etc.
+
+## Testing Guidelines
+
+### Test Structure
+- Use classes for all tests
+- Test file names end with `Test`
+- One test file per implementation file
+- Use anyio for async tests (not asyncio)
+
+### Test Quality (NO WEAK TESTS!)
+- No `if` statements in tests
+- Test actual values, not just shapes
+- No try/catch for testing
+- Use hardcoded expected values
+- No magic numbers - use class constants
+
+### Running Tests
+```bash
+poe test                         # All tests
+poe test-cov                     # With coverage report
+poe test-cov-check              # Enforce 85% coverage
+```
+
+## Git Workflow
+- Check `git status` before commits
+- Never commit unless explicitly asked
+- Run `poe check` before any commits
+- Keep changes focused and minimal
+
+## Performance Expectations
+- Initialization: < 0.5 seconds
+- Search operations: < 0.5 seconds  
+- Pickle load/save: < 0.5 seconds

.serena/memories/project_overview.md

@@ -0,0 +1,33 @@
+# Catalyst Project Overview
+
+## Purpose
+Catalyst is an enterprise-grade toolkit for building LLM-powered document processing applications. It turns complex documents into queryable knowledge systems designed specifically for regulated industries where accuracy and source attribution are critical.
+
+## Key Features
+- **Hybrid Intelligence**: Combines vector search + keyword extraction + relationship mapping
+- **Knowledge Graphs**: Extracts acronyms and builds relationships between terms
+- **Structure Preservation**: Tables, images, and document hierarchy stay intact
+- **Source Attribution**: Every answer includes exact page numbers and document sections
+- **Zero Dependencies**: Fully self-contained with embedded models (~32MB)
+- **Offline Capable**: No external API calls required
+- **Async Processing**: Built on ASGI for high-performance document pipelines
+
+## Target Use Cases
+- Regulatory compliance document analysis
+- Legal due diligence across contracts
+- Financial analysis with risk metrics
+- Technical documentation understanding
+- Audit support with complete evidence chains
+
+## Tech Stack
+- **Language**: Python 3.12+
+- **Package Management**: uv (NOT pip)
+- **Core Dependencies**:
+  - langchain-core>=0.3.75
+  - model2vec>=0.6.0 (embedded model)
+  - numpy, tenacity, trio, sqlalchemy
+- **Optional Dependencies**:
+  - API: fastapi, fastmcp, agno, rapidfuzz
+  - Extraction: pdfplumber, scikit-learn, pypdf, pyoxipng
+- **Testing**: pytest, anyio (preferred over asyncio)
+- **Development**: mypy, ruff, black, isort, poethepoet

.serena/memories/project_structure.md

@@ -0,0 +1,56 @@
+# Project Structure
+
+## Main Directories
+
+```
+com_blockether_catalyst/
+├── src/com_blockether_catalyst/           # Main source code
+│   ├── asgi/                              # ASGI web application modules
+│   ├── consensus/                         # LLM consensus and voting systems
+│   ├── encoder/                           # Text encoding and embeddings
+│   ├── knowledge/                         # Core knowledge extraction & search
+│   ├── prompt/                            # Prompt engineering and alignment
+│   ├── integrations/                      # External system integrations
+│   │   └── agno/                         # Agno workflow engine integration  
+│   ├── utils/                            # Shared utilities
+│   └── assets/                           # Static assets (models, etc.)
+├── tests/                                # Test suite (mirrors src structure)
+├── tools/                                # Development and utility scripts
+├── docs/                                 # Documentation
+└── verification/                         # Deployment verification scripts (Python only)
+```
+
+## Key Modules
+
+### Knowledge System (`knowledge/`)
+- **KnowledgeSearchCore.py**: Main search engine with hybrid vector+keyword search
+- **KnowledgeExtractionCore.py**: Document processing and term extraction
+- **PDFKnowledgeExtractor.py**: PDF-specific extraction logic
+- **KnowledgeVisualizationASGIModule.py**: Web UI for knowledge exploration
+
+### Consensus System (`consensus/`)  
+- **Consensus.py**: Multi-LLM consensus mechanism
+- **ConsensusCore.py**: Core consensus algorithms
+- **VotingComparison.py**: Voting strategies for LLM outputs
+
+### Prompt System (`prompt/`)
+- **PromptAlignmentCore.py**: Prompt optimization and alignment
+- **PrincipleBasedAlignmentStrategy.py**: Alignment strategies
+
+### Utils (`utils/`)
+- **TypedCalls.py**: Type-safe LLM API calls
+- **ConcurrentProcessor.py**: Async processing utilities
+- **instructor/**: Structured LLM output handling
+
+## Naming Conventions
+- Core implementation files end with `Core` (e.g., `KnowledgeSearchCore.py`)
+- Test files end with `Test` (e.g., `KnowledgeSearchCoreTest.py`)
+- Type definition files end with `Types` (e.g., `KnowledgeExtractionTypes.py`)
+- ASGI modules end with `ASGIModule` for web components
+
+## File Organization Pattern
+Each major module typically has:
+- `{Module}Core.py` - Main implementation
+- `{Module}Types.py` - Type definitions and data models  
+- `{Module}Test.py` - Comprehensive test suite
+- `internal/` subfolder for implementation details (where applicable)

.serena/memories/suggested_commands.md

@@ -0,0 +1,68 @@
+# Essential Development Commands
+
+## Package Management (ALWAYS use uv)
+```bash
+# Installation
+uv add package
+
+# Running tools  
+uv run tool
+
+# Running examples
+uv run python3 examples/*
+
+# Running verification scripts
+uv run python3 verification/*
+
+# Upgrading packages
+uv add --dev package --upgrade-package package
+
+# FORBIDDEN: uv pip install, @latest syntax
+```
+
+## Code Quality & Testing
+```bash
+# Linting and type checking
+poe verify          # Runs both lint and typecheck
+poe lint           # uv run ruff check src/ tests/
+poe typecheck      # uv run mypy src/ tests/
+
+# Formatting
+poe format         # Runs ruff, black, and isort formatters
+poe format-ruff    # uv run ruff format src/ tests/
+poe format-black   # uv run black src/ tests/
+poe format-isort   # uv run isort src/ tests/
+
+# Testing
+poe test           # uv run python3 -m pytest
+poe test-cov       # uv run python3 -m pytest --cov=src --cov-report=html
+poe test-cov-check # uv run python3 -m pytest --cov=src --cov-report=term-missing --cov-fail-under=85
+
+# Complete workflow
+poe check          # format + verify + test-cov-check + check-docs
+```
+
+## Documentation
+```bash
+poe docs-serve     # uv run mkdocs serve
+poe docs-build     # uv run mkdocs build  
+poe check-docs     # Validates documentation structure
+```
+
+## Cleaning
+```bash
+poe clean          # Cleans pyc files and cache
+poe clean-cache    # uv cache clean
+poe clean-pyc      # Removes __pycache__ and .pyc files
+```
+
+## System Commands (macOS Darwin)
+```bash
+# Standard Unix commands available:
+ls, cd, grep, find, git, etc.
+```
+
+## Important Notes
+- ALWAYS run `poe format` before `poe verify`
+- If pytest fails with asyncio marks, try: `PYTEST_DISABLE_PLUGIN_AUTOLOAD="" uv run --frozen pytest`
+- NO shell scripts (.sh) allowed in verification/ directory - use Python scripts only

.serena/memories/task_completion_checklist.md

@@ -0,0 +1,43 @@
+# Task Completion Checklist
+
+When completing any development task, follow this exact order:
+
+## 1. Code Quality Checks (REQUIRED)
+```bash
+# Always run in this order:
+poe format      # Format code first
+poe verify      # Then lint and typecheck  
+```
+
+## 2. Testing (REQUIRED)  
+```bash
+poe test        # Run all tests
+# OR for coverage check:
+poe test-cov-check  # Ensures >=85% coverage
+```
+
+## 3. Full Verification (RECOMMENDED)
+```bash
+poe check       # Runs format + verify + test-cov-check + check-docs
+```
+
+## Error Resolution Priority
+1. **Formatting issues first** (line length, imports, etc.)
+2. **Type errors second** (add type hints, None checks, function signatures)
+3. **Linting issues last** (unused imports, etc.)
+
+## Common Fixes
+- **Line length**: Break strings with parentheses, multi-line function calls, split imports
+- **Type errors**: Add None checks, narrow string types, match existing patterns
+- **Missing dependencies**: Check package.json/pyproject.toml for available libraries
+
+## Pre-commit Requirements
+- Check git status before commits
+- Ensure all linting/typing passes
+- Keep changes minimal and focused
+- Document public APIs
+- Test thoroughly, especially edge cases and error conditions
+
+## NEVER commit changes unless explicitly asked
+- Only commit when user explicitly requests it
+- Run all quality checks before committing