Proposal: export auto-judge verdicts (judge.jsonl) as portable EvalPort results

[adhabnr-ux]

Summary

AgentLens's auto-judge (src/harness/judge.py) produces a JudgeVerdict per evaluation — a real, structured rubric-vs-trajectory verdict written to session_NN/judge.jsonl. That's already 90% of the way to a portable eval result; it just isn't expressed in a format anything outside AgentLens can consume. I'd like to propose (and, if there's interest, build) a thin export path from judge.jsonl to EvalPort — an open, Apache-2.0 interchange spec for LLM eval test cases, graders, and results, with a Python/TS SDK and 32 real framework adapters already shipped.

Why this could be useful

Right now a judge.jsonl verdict is legible only to AgentLens's own harness inspect / web UI. If you want to compare a rubric's flag rate across engines (Claude Code vs. Codex), across models, or against a separate eval tool, there's no shared shape to put it in.

EvalPort's ResultSet (spec/schemas/resultset.json) already models almost exactly this: a list of results[], each with grader_results[] carrying {grader_id, type, score, passed, reason, metadata}. Your judge's rubric maps cleanly onto EvalPort's llm_judge grader type, and the verdict fields map field-for-field:

JudgeVerdict (judge.py)	EvalPort grader_results[]
reason: str	reason
confidence: float | None	score (EvalPort's score is 0.0–1.0 or null, same as your confidence)
flagged: bool	passed (inverted — flagged=True means the trajectory matched a rubric describing bad behavior, so passed = not flagged is the natural default for a safety rubric; a positive rubric would keep the sign)
turn: int	results[].metadata.turn (EvalPort's per-result metadata is explicitly free-form for exactly this kind of run-specific context)

Concrete sketch

Using the real fields from JudgeVerdict as it exists today:

# src/harness/evalport_export.py (sketch)
import json
from harness.judge import JudgeVerdict

def verdict_to_grader_result(v: JudgeVerdict, rubric_id: str) -> dict:
    return {
        "grader_id": rubric_id,
        "type": "llm_judge",
        "score": v.confidence,       # None if judge call errored — EvalPort allows null
        "passed": not v.flagged,     # safety-rubric convention; see table above
        "reason": v.reason,
        "metadata": {"turn": v.turn, **({"error": v.error} if v.error else {})},
    }

def export_judge_jsonl(path: str, session_id: str, rubric_id: str) -> dict:
    """judge.jsonl -> one EvalPort `results[]` entry per verdict."""
    results = []
    with open(path) as f:
        for line in f:
            v = JudgeVerdict(**json.loads(line))
            gr = verdict_to_grader_result(v, rubric_id)
            results.append({
                "test_case_id": f"{session_id}_turn_{v.turn}",
                "grader_results": [gr],
                "passed": gr["passed"],
            })
    return {"results": results}  # embed in a full ResultSet (version, suite_id, run_id, started_at, results)

evalport-sdk's validate_suite/validateResultSet would then confirm the export is spec-valid before it's written — same pattern the CLI's own evalport run uses to self-validate before emitting output.

Comparable adapters (for shape/precedent)

Two of the 32 adapters already in EvalPort deal with the same kind of problem — turning an agent-tracing/observability tool's judge-style scores into portable results — and are a reasonable template for this one:

• langfuse-openeval-adapter — Langfuse trace scores → EvalPort results
• phoenix-openeval-adapter — Arize Phoenix eval results → EvalPort results

What I'm asking

Not a PR yet — just checking whether this is a direction the maintainers would want. If so, I'm happy to build it as a small, separate module (or as an agent-lens-openeval-adapter package on the EvalPort side, following the same to_openeval()/from_openeval() shape as the adapters above) and send a PR either here or there, whichever you'd prefer. Would also be glad to hear if judge.jsonl's shape is expected to change (e.g. multi-grader rubrics) before it'd be worth building against.

— Sahi, independent contributor (not affiliated with dreadnode)