Add model checks for pytest

[planetf1]

When running tests it's easy to get failures by not having the correct models installed. Errors may be hidden in the large output log.

We should look at adding classifiers in tests accordingly and reporting errors at collection time, or another validation mechanism

Ollama will be the most common, but we also should consider vllm and other backends.

A lighter option is to document bit I think a programmatic check is more robust and simpler to understand

[psschwei]

Would also be good to include a section in docs (either CONTRIBUTING.md or test/README.md) around how to set up local environment for running tests.

[psschwei]

Possible starting point for doc on setting up environment to run tests

Models Required to Run Tests in test/

Based on analysis of the test suite, here are all the models you need organized by backend:

Ollama Backend

• granite4:micro-h (IBM_GRANITE_4_HYBRID_MICRO) - Primary test model
• granite3.2-vision - Vision model tests

HuggingFace Backend (Local)

• ibm-granite/granite-3.3-8b-instruct - Main HF tests (requires GPU + 48GB+ RAM)
• mistralai/Mistral-7B-Instruct-v0.3 - Tool calling tests

vLLM Backend (Local)

• Qwen/Qwen3-0.6B (QWEN3_0_6B) - Main vLLM tests
• mistralai/Mistral-7B-Instruct-v0.3 - Tool calling tests
• ibm-granite/granite-4.0-micro - OpenAI-compatible vLLM tests

Watsonx Backend (API)

• ibm/granite-4-h-small (IBM_GRANITE_4_HYBRID_SMALL) - Requires Watsonx API credentials

Bedrock Backend (API)

• openai.gpt-oss-20b (OPENAI_GPT_OSS_20B) - Requires AWS Bedrock credentials
• openai.gpt-oss-120b (OPENAI_GPT_OSS_120B) - Requires AWS Bedrock credentials

LiteLLM Backend

• Uses same models as above through LiteLLM proxy:
  • ollama_chat/granite4:micro-h
  • watsonx/ibm/granite-4-h-small

Notes:

• Ollama models are the most commonly used across tests (lightweight, local)
• HuggingFace and vLLM tests require GPU and significant RAM (48GB+)
• Watsonx tests require: WATSONX_API_KEY, WATSONX_URL, WATSONX_PROJECT_ID
• Bedrock tests require: AWS_BEARER_TOKEN_BEDROCK
• Most tests are marked as qualitative and can be skipped in CI with CICD=1
• To run minimal tests: uv run pytest -m "not qualitative" (uses only Ollama)

Quick Start for Running Tests

Minimal Setup (Ollama only)

# Install and start Ollama
ollama serve

# Pull required models
ollama pull granite4:micro-h
ollama pull granite3.2-vision

# Run fast tests only
uv run pytest -m "not qualitative"

Full Local Setup (with GPU)

# Install all dependencies
uv sync --all-extras --all-groups

# Pull Ollama models
ollama pull granite4:micro-h
ollama pull granite3.2-vision

# HuggingFace models will be downloaded automatically when tests run
# Requires: GPU with 48GB+ RAM

# Run all tests (including slow ones)
uv run pytest

API-based Backends

Set environment variables before running tests:

Watsonx:

export WATSONX_API_KEY="your-key"
export WATSONX_URL="your-url"
export WATSONX_PROJECT_ID="your-project-id"

Bedrock:

export AWS_BEARER_TOKEN_BEDROCK="your-token"

[planetf1]

I think getting the docs in is a great first step. I would make clear that the api based backends are optional - the framework aims to run what you have configured. (I actually disabled my watsonx setup ie NULLed out my env vars, as my account doesn't have enough capability)

[code4days]

I can update the docs for the local ollama only part. I'll submit a PR once I get the tests to pass.