metamorphic-eval

Metamorphic testing library for AI agent evals.

Catch inconsistencies in agent behavior — no ground-truth labels required.

---

The Idea

Traditional evals need a correct answer to compare against. Metamorphic testing doesn't. Instead, it defines relations that must hold between outputs when you transform the input in a semantically equivalent way.

"If I rephrase this question, the agent should give the same answer." "If I shuffle these options, the agent shouldn't suddenly prefer a different one."

When a relation is violated, you've found a bug — even without knowing what the "right" answer is.

---

Quickstart

from metamorphic_eval import Suite
from metamorphic_eval.transformers import Paraphrase, AddNoise, ReorderItems
from metamorphic_eval.relations import Equivalent, OrderInvariant
from metamorphic_eval.comparators import LLMJudge, StructuredMatch

suite = Suite(agent=my_agent)

suite.add(
    name="rephrasing_robustness",
    baseline={"question": "What caused the 2008 financial crisis?"},
    transform=Paraphrase(model="claude-sonnet-5"),
    relation=Equivalent(comparator=LLMJudge(model="claude-sonnet-5")),
    n_variants=5,
)

suite.add(
    name="ordering_bias",
    baseline={"question": "Which option is best?", "options": ["A", "B", "C"]},
    transform=ReorderItems(field="options"),
    relation=OrderInvariant(comparator=StructuredMatch(field="choice")),
)

report = suite.run()
report.summary()

---

Core Concepts

Transformers

Mutate an input while preserving its semantic meaning.

Transformer	What it does
Paraphrase	LLM rewrites the question differently
AddNoise	Typos, casing, punctuation variation
ReorderItems	Shuffles list fields
TranslateRoundtrip	EN → another language → EN
ChangeFormality	Casual ↔ formal register
AddIrrelevantContext	Injects a sentence that shouldn't affect the answer

Metamorphic Relations

Expected relationship between the original and transformed output.

Relation	Meaning
Equivalent	Outputs should be semantically the same
OrderInvariant	Output shouldn't change when input order changes
MonotoneIncrease	Output score should increase with a given signal

Comparators

Strategy for deciding whether two outputs satisfy a relation.

Comparator	Notes
LLMJudge	Ask a model if outputs agree. Most flexible, costs tokens.
SemanticSimilarity	Embedding cosine sim with a threshold. Fast, less precise.
StructuredMatch	JSON field-by-field comparison. Precise but requires structured output.

---

What Gets Caught

Failure mode	Suite configuration
Prompt sensitivity	Paraphrase → Equivalent
Position/ordering bias	ReorderItems → OrderInvariant
Hallucination inconsistency	Paraphrase → StructuredMatch
Noise robustness	AddNoise → Equivalent
Context bleed	AddIrrelevantContext → Equivalent

---

Install

pip install metamorphic-eval

Or from source:

git clone https://github.com/integralquality/metamorphic-eval
cd metamorphic-eval
pip install -e ".[dev]"

---

Design Philosophy

• Upstream of evals — this library generates variants and surfaces violations. Scoring and pass/fail thresholds are your eval framework's job.
• Agent interface is a plain callable — agent: Callable[[dict], Any]. No SDK dependency.
• No ground truth required — consistency checking works without labeled data.

---

Status

Early development. API is unstable.