Skip to content

Add Python data loading library for IMO Bench datasets with testing#5

Open
tryh4rd-26 wants to merge 2 commits intogoogle-deepmind:mainfrom
tryh4rd-26:main
Open

Add Python data loading library for IMO Bench datasets with testing#5
tryh4rd-26 wants to merge 2 commits intogoogle-deepmind:mainfrom
tryh4rd-26:main

Conversation

@tryh4rd-26
Copy link

I built a Python library to make working with IMO Bench way easier since right now everyone has to manually parse CSVs which is kind of a pain. The library gives you a clean API where you can just call load_answerbench(category="Algebra" and get back 100 type-safe problem objects instead of messing with csv.DictReader.

I added filtering for all the obvious things (category, difficulty, source, points), proper lazy loading for the big GradingBench file so it doesn't eat all your memory, and validation.

Everything is tested (35 tests all passing with the actual CSV files), uses dataclasses so your IDE knows what fields exist, and auto-finds the data directory so it just works out of the box.

This should unblock people who want to build evaluation pipelines or analyze the datasets without writing their own CSV parsers, and it's ready to publish to PyPI if we want. No breaking changes since it's all new code.

@tryh4rd-26
Copy link
Author

@ychervonyi

@Ashutosh0x
Copy link

I've tested the library in a Python 3.8 environment and found that the 'list[]' and 'dict[]' type hints cause errors. I've implemented compatibility fixes using 'typing.List' and 'typing.Dict' in my dependent PR #6. Would highly recommend incorporating those fixes into this PR to support older Python versions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants