Hi — I maintain EvalPort (Apache 2.0), an open JSON spec for portable LLM evaluation datasets (test cases, graders, suites, results) designed to move between eval frameworks without losing meaning.
I looked at notebooks/evals/ before writing this — you already have ragas-evaluation.ipynb, the embeddings-eval notebooks (openai-embeddings-eval.ipynb, voyageai-embeddings-eval.ipynb, angle-embeddings-eval.ipynb, gemini-embeddings-eval.ipynb), and Patronus_MongoDB.ipynb, all indexed in that folder's README.md. That's a genuinely good fit for a small companion notebook: take the query/ground-truth pairs one of those notebooks already evaluates (e.g. the Ragas one) and show them exported as a portable EvalPort suite + ResultSet, so the same eval data is usable outside Ragas too.
Rough shape:
from openeval.validate import validate_suite, validate_result_set
suite = {
"version": "1.0.0",
"id": "mongodb-rag-eval",
"graders": [{"id": "gr1", "type": "exact_match"}],
"test_cases": [
{"id": "tc1", "input": "...", "expected_output": "...", "graders": ["gr1"]}
],
}
assert validate_suite(suite).valid
# ... run against the MongoDB Atlas-backed pipeline, grade, build a ResultSet ...Would a notebook like this be a welcome addition to notebooks/evals/, listed alongside the others in that folder's README? Keeping the ask scoped to one self-contained notebook per your CONTRIBUTING.md norms — no pressure if this isn't the right fit right now.
Spec: https://github.com/adhabnr-ux/evalport/blob/main/spec/SPEC.md
Hi — I maintain EvalPort (Apache 2.0), an open JSON spec for portable LLM evaluation datasets (test cases, graders, suites, results) designed to move between eval frameworks without losing meaning.
I looked at
notebooks/evals/before writing this — you already haveragas-evaluation.ipynb, the embeddings-eval notebooks (openai-embeddings-eval.ipynb,voyageai-embeddings-eval.ipynb,angle-embeddings-eval.ipynb,gemini-embeddings-eval.ipynb), andPatronus_MongoDB.ipynb, all indexed in that folder'sREADME.md. That's a genuinely good fit for a small companion notebook: take the query/ground-truth pairs one of those notebooks already evaluates (e.g. the Ragas one) and show them exported as a portable EvalPort suite + ResultSet, so the same eval data is usable outside Ragas too.Rough shape:
Would a notebook like this be a welcome addition to
notebooks/evals/, listed alongside the others in that folder's README? Keeping the ask scoped to one self-contained notebook per your CONTRIBUTING.md norms — no pressure if this isn't the right fit right now.Spec: https://github.com/adhabnr-ux/evalport/blob/main/spec/SPEC.md