Skip to content

Add create from generator#88

Merged
jaclark5 merged 1 commit intomainfrom
ds_from_gen
Apr 7, 2026
Merged

Add create from generator#88
jaclark5 merged 1 commit intomainfrom
ds_from_gen

Conversation

@jaclark5
Copy link
Copy Markdown
Collaborator

Description

Currently creating a dataset involves a list[dict] that is imported all at once into descent.targets.energy.create_dataset(). This requires that the entire dataset is held in memory.

HuggingFace dataset can be created with generators using Dataset.from_generator() but the Dataset is abstracted away in descent.

This PR adds a function descent.targets.energy.create_dataset_from_generator() to regain this capability from the datasets library.

Marking as a draft until I successfully use this with my current work, just in case I run into an issue

Status

  • Ready to go

@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented Mar 13, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.49%. Comparing base (40d4051) to head (3ab21eb).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main      #88   +/-   ##
=======================================
  Coverage   99.49%   99.49%           
=======================================
  Files          11       11           
  Lines         981      997   +16     
=======================================
+ Hits          976      992   +16     
  Misses          5        5           
Flag Coverage Δ
unittests 99.49% <100.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@j-wags j-wags marked this pull request as ready for review April 6, 2026 23:34
@j-wags j-wags assigned lilyminium and unassigned jaclark5 Apr 6, 2026
Copy link
Copy Markdown
Contributor

@lilyminium lilyminium left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM thanks @jaclark5 !

@jaclark5 jaclark5 merged commit d241a5a into main Apr 7, 2026
1 check passed
@jaclark5 jaclark5 deleted the ds_from_gen branch April 7, 2026 02:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants