Skip to content
BranchesTags

Repository files navigation

BioPrep-AI

A CLI tool and Streamlit web UI that takes a raw biosensor file (signal or image) plus a plain-English description of the sensor and goal, audits it (deterministic math for signals, vision model + CV fallback for images), then generates, self-tests, and containerizes a custom preprocessing pipeline — with a self-healing retry loop if generated code fails.

DOI

Built with Codex & GPT-5.6

Built for OpenAI Build Week 2026, using OpenAI's models in two distinct roles:

Development — Codex was used throughout development to generate and refine project files and code, including scaffolding the CLI structure and generated- pipeline templates, refactoring the sandboxing and self-test generation flow, and sharpening ideas during iteration (tightening error handling, fixing test isolation issues, improving the audit-report generator).

Runtime — GPT-5.6 is supported as one of several interchangeable, OpenAI- compatible LLM providers (see "LLM providers" below). When configured, it is used for interpreting biosensor files during audit (vision-based image analysis, signal reasoning), selecting preprocessing stages based on the sensor description and audit results, and explaining quality-impact metrics in plain language. If no LLM is configured, or a call fails/times out, the tool falls back to deterministic, offline-safe logic automatically.

Requirements

Requirement Version
Python 3.11 or higher
OS Windows, macOS, or Linux
Disk space ~50 MB (dependencies)
Internet Only needed for LLM providers (optional)

Python packages (installed automatically)

Core (always installed):

  • numpy >= 1.26 — array math
  • scipy >= 1.11 — signal processing (bandpass, PSD, Savitzky-Golay)
  • opencv-python-headless >= 4.8 — image processing
  • pillow >= 10.0 — image I/O
  • typer >= 0.12 — CLI framework
  • pytest >= 8.0 — test runner
  • rich >= 13.0 — styled terminal output

Optional — LLM support:

  • openai >= 1.40 — enables LLM-powered stage selection and metric annotations

Optional — Web UI:

  • streamlit >= 1.30 — browser-based interface with file upload, preview, and downloads

Optional — file watcher:

  • watchdog >= 4.0 — efficient file change detection for watch mode

Installation (step by step)

1. Clone the repo

git clone https://github.com/KharfiIslam/BioPrep-AI.git
cd BioPrep-AI-main

2. Check your Python version

python --version
# Must show 3.11 or higher

If you don't have Python 3.11+, download it from https://www.python.org/downloads/

3. Install all dependencies

Windows (double-click run.bat): A launcher menu will appear — pick option 3 to install, then 1 or 2 to run.

Or run from terminal:

# Install everything (core + LLM + web UI):
python -m pip install -e ".[ui,llm]"

# Or install only core (no LLM, no web UI):
python -m pip install -e .

# Or use requirements.txt directly:
python -m pip install -r requirements.txt

4. Generate sample files (optional)

python scripts/make_samples.py

This creates demo files in samples/ (EEG CSV, ECG NPY, lateral-flow JPG).

5. Verify installation

python -m pytest tests/ -v
# Should show all tests passing

How to run

Option A: Launcher script (easiest)

Windows: Double-click run.bat — a menu appears:

============================================
   preprocess-ops launcher
============================================

  Choose an option:

    1) CLI menu (terminal)
    2) Web UI (browser)
    3) Install / update dependencies
    4) Run tests
    5) Quit

  Enter number (1-5):

Pick 1 for the terminal menu, 2 for the browser UI.

Linux / macOS:

chmod +x run.sh
./run.sh

Option B: CLI menu (interactive)

python -m preprocess_ops

A rich terminal menu guides you through:

  1. Audit a file
  2. Generate a pipeline (audit + sandbox + self-heal)
  3. Full demo (generate → test → build → report)
  4. Run tests
  5. Build Dockerfile
  6. Configure LLM
  7. Launch web UI
  8. Quit

Option C: CLI one-shot commands

# Audit a signal file:
python -m preprocess_ops audit samples/eeg_sample.csv \
  --describe "3-electrode dry EEG, want it clean for seizure classification"

# Generate a pipeline (offline, no LLM):
python -m preprocess_ops generate samples/eeg_sample.csv \
  --describe "dry EEG cleanup" --offline

# Generate with LLM + full report:
python -m preprocess_ops generate samples/lateral_flow_strip.jpg \
  --describe "lateral flow strip quantification" --offline

# Run all tests:
python -m preprocess_ops test

# Build Docker image:
python -m preprocess_ops build --run

# Batch process a directory:
python -m preprocess_ops batch samples/ --describe "batch cleanup" --offline

# Watch for file changes:
python -m preprocess_ops watch samples/eeg_sample.csv --poll 2

Option D: Web UI (browser)

python -m streamlit run preprocess_ops/ui/streamlit_app.py

Opens http://localhost:8501 with four pages:

Page What it does
Home Overview, feature summary, LLM status
Pipeline Upload file → preview → configure → run → download artifacts
LLM Config Set up provider, API key, model — save for reuse
Batch Upload multiple files, process all at once

What the tool produces

Every pipeline run creates these files in output/:

File Description
profile.json Audit results (shape, sampling rate, PSD peaks, lighting, blur, ROI)
pipeline.py Generated, inspectable, editable preprocessing script
cleaned.npy / cleaned.png Cleaned output (signal or image)
cleaned.metrics.json ROI coordinates or intensity metrics (images only)
quality_comparison.json Before/after quality deltas with percent changes
audit_report.html One-page HTML report bundling everything
test_generated_pipeline.py Pytest contract test for the generated pipeline
test_result.json Test pass/fail status and pytest output
generation.json Pipeline metadata (attempts, fallback status, sandbox transcript)

Quality metrics

Every run computes before/after quality deltas:

╭──────────── QUALITY IMPACT ────────────╮
│  Noise (raw)      0.412                │
│  Noise (cleaned)  0.087   ▼ 78.9%      │
│  Sharpness (raw)  152.7                │
│  Sharpness (clean) 210.3   ▲ 37.7%     │
╰─────────────────────────────────────────╯

Metric annotations explain why each metric changed:

  • LLM-generated — configured model explains each delta in plain English
  • Rule-based — deterministic fallback using a stage-effect knowledge base
  • Mixed — LLM explains some metrics, rule-based covers the rest

Audit report

The HTML report bundles everything into a single shareable document:

  1. Audit summary — modality, profiler, shape, sampling rate / lighting / blur / ROI
  2. Quality impact — before/after metrics with percent deltas
  3. Why metrics changed — LLM or rule-based annotations per metric
  4. Sandbox result — pipeline source, attempts, transcript
  5. Generated pipeline — full syntax-highlighted code
  6. Test results — PASSED/FAILED with pytest output
  7. Artifacts — file paths

Generate it with:

python -m preprocess_ops report --output-dir output/

LLM providers (optional)

Install LLM support:

python -m pip install -e ".[llm]"
Provider Setup
OpenAI PREPROCESS_OPS_PROVIDER=openai + OPENAI_API_KEY=...
Groq PREPROCESS_OPS_PROVIDER=groq + GROQ_API_KEY=...
OpenRouter openrouter + key — Kimi, GLM, and many others
Kimi / Moonshot PREPROCESS_OPS_PROVIDER=kimi + API key
GLM / Zhipu PREPROCESS_OPS_PROVIDER=glm + API key
DeepSeek PREPROCESS_OPS_PROVIDER=deepseek + API key
Together together + TOGETHER_API_KEY=...
Ollama (local) PREPROCESS_OPS_PROVIDER=ollama
LM Studio (local) PREPROCESS_OPS_PROVIDER=lmstudio
Other PREPROCESS_OPS_PROVIDER=other + PREPROCESS_OPS_BASE_URL=...

Configure via CLI menu (option 6), web UI (LLM Config page), or environment variables. Settings can be saved to .preprocess_ops_llm.json for reuse.

CLI reference

Command Purpose
python -m preprocess_ops / menu Interactive numbered menu
audit FILE Write output/profile.json, print audit summary
generate FILE Audit + synthesize + sandbox + cleaned data + test + report
report Bundle artifacts into audit_report.html
batch DIR Process every supported file in a directory
watch FILE Re-run on file changes (Ctrl-C to stop)
test Run unit tests + generated contract test
build [--run] Render Dockerfile, optionally docker build
web Launch Streamlit UI

Common flags: --describe/-d, --sampling-rate/-r, --offline, --output-dir/-o, --run-id, --roi "x,y,w,h", --json, -v/--verbose.

Architecture

User input: file (CSV/NumPy or PNG/JPG) + free-text description
                    ↓
        [Modality Dispatcher] — routes by file type
        ↓                                    ↓
  [Signal Profiler]                   [Image Profiler]
  deterministic (scipy)               vision model call
  sampling rate, PSD peaks,           → on fail/timeout →
  amplitude, noise floor              CV fallback (OpenCV)
        ↓                                    ↓
              profile.json (unified schema)
                    ↓
        [Pipeline Synthesizer] (any OpenAI-compatible LLM)
        constrained stage plan inside fixed class template
                    ↓
        [Sandbox Execution + Self-Heal]
        run on sample → error/NaN? → retry once → trusted fallback
                    ↓
        [Quality Metrics] — before/after noise, RMS, contrast, sharpness
                    ↓
        [Metric Annotations] — LLM or rule-based explanations
                    ↓
        [Test Generator] → pytest contract test
                    ↓
        [Audit Report] → single HTML page
                    ↓
        [Containerizer] → Dockerfile

Repo layout

preprocess-ops/
├── run.bat                        # Windows launcher (double-click)
├── run.sh                         # Linux/macOS launcher
├── requirements.txt               # Dependency list
├── preprocess_ops/                # Python package
│   ├── cli.py                     # Typer CLI (menu + subcommands)
│   ├── service.py                 # Shared orchestration (run_pipeline)
│   ├── metrics.py                 # Before/after quality proxies
│   ├── annotations.py             # LLM + rule-based metric explanations
│   ├── report.py                  # HTML audit report renderer
│   ├── present.py                 # Rich terminal UI helpers
│   ├── llm.py                     # OpenAI-compatible LLM client
│   ├── audit/                     # Input file profiling
│   ├── filters/                   # Trusted preprocessing stages
│   ├── synth/                     # Pipeline code generation
│   ├── testgen/                   # Contract test generation
│   └── ui/                        # Streamlit web UI
│       └── streamlit_app.py       # 4-page interface
├── tests/                         # Test suite
├── samples/                       # Demo input files
├── docker/Dockerfile.template
├── scripts/make_samples.py
├── output/                        # Generated artifacts
└── pyproject.toml

Design principles

  • Deterministic where precision matters. Signal audit is pure scipy; never LLM-guessed numbers.
  • Scaffolded generation. The model chooses stages from a trusted whitelist; method bodies stay inside a fixed class template.
  • Self-healing. Sandbox-run every pipeline → one retry with error context → trusted fallback.
  • Same contract across modalities. Both profilers emit a shared profile.json shape.
  • Quantified proof. Every run produces before/after quality metrics with percent deltas.
  • Transparent explanations. Metric annotations show source (LLM / rule-based / mixed).
  • Provenance by default. HTML report bundles everything into a single shareable document.

Quick troubleshooting

Problem Fix
python not found Use python3 instead, or add Python to PATH during install
pip not found Use python -m pip instead
ModuleNotFoundError Run python -m pip install -e ".[ui,llm]"
Streamlit won't start Run python -m pip install streamlit>=1.30
LLM not connecting Check provider URL and API key, or use --offline
Tests fail Run python -m pip install -e . first to install the package

About

LLM-driven biosensor preprocessing: audit signal/image files, generate sandboxed pipelines, and containerize with offline-safe fallbacks.

Topics

biosensorcodexcomputer-visiondockerllmopenaipreprocessingpythonsignal-processingstreamlit

Resources

Readme
MIT license
Activity

Stars

1 star

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages