Skip to content

Add machine-readable StructEval discovery index #7

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
reacher-z wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Add machine-readable StructEval discovery index #7

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions llms.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# StructEval

> StructEval is an open-source benchmark and evaluation framework for testing language models' ability to generate structured outputs, including renderable and non-renderable formats.

StructEval supports inference, rendering, and evaluation through a command-line workflow. It evaluates generated outputs with structural validation and visual checks for renderable artifacts.

## Canonical resources

- [GitHub repository](https://github.com/TIGER-AI-Lab/StructEval): source code, CLI, examples, and dataset files.
- [Research paper](https://arxiv.org/abs/2505.20139): *StructEval: Benchmarking LLMs' Capabilities to Generate Structural Outputs*.
- [TMLR venue](https://openreview.net/forum?id=3w6M6fJ6fN): peer-reviewed publication record.
- [Dataset directory](https://github.com/TIGER-AI-Lab/StructEval/tree/main/dataset): renderable, non-renderable, and consolidated StructEval data files.
- [Citation record](https://github.com/TIGER-AI-Lab/StructEval#citation): BibTeX citation and author list.

## Evaluation capabilities

- Generate model outputs from structured-output prompts.
- Render outputs that require visual evaluation with Playwright.
- Validate keys and raw output metrics for non-renderable formats.
- Score render quality with visual question answering for renderable outputs.
- Produce per-example metrics and an overall `final_eval_score` between 0 and 1.

## Citation

If you use StructEval, cite the paper and link to the canonical repository above.