Skip to content

Check for many contributions to warn of potential OOM and clarify use…#1119

Draft
bfoley12 wants to merge 2 commits into
materialsproject:mainfrom
bfoley12:bulk-contrib-docs
Draft

Check for many contributions to warn of potential OOM and clarify use…#1119
bfoley12 wants to merge 2 commits into
materialsproject:mainfrom
bfoley12:bulk-contrib-docs

Conversation

@bfoley12

@bfoley12 bfoley12 commented Jul 7, 2026

Copy link
Copy Markdown

Summary

Leo found that our submit_contributions client method needed to be clearer in documentation.

  1. No recommended batch size for submit_contributions. On a big dataset it's easy to OOM, since the whole batch gets held in memory on the client side. I hit an out-of-memory kill trying to submit all ~16.5k at once, and fixed it just by chunking into batches of 500. Might be worth a note in the docstring about a sane batch size, or a warning when the batch is large.
  2. A single duplicate aborts the entire upload. If any component is a duplicate, submit_contributions raises and kills the whole batch. In my case a handful of my systems relaxed to byte-identical structures (different starting configs, same DFT minimum), which is legit data — so the whole run died on it. Would be nicer to skip/warn on dupes and keep going by default.
    For reference (mpcontribs-client init.py):

the raise is at line 2345, guarded byif not ignore_dupes and dupe:at 2343
there's even a# TODO add matching duplicate info to msgat 2344 — the message doesn't say which contribution it collides with, which made it a bit of a hunt to track down
I did findignore_dupes/skip_dupe_check(params at 2120/2122) which solve it, but they weren't obvious — maybe worth surfacing in the docs, or flipping skip-and-warn to the default

Changes

  • Added paragraph to docstring warning about the need to batch due to potential OOM errors
  • Log warning if contribs list is larger than an arbitrary number (I chose 5,000)
    • This could be based on byte-size of the contribs list[dict] that gets passed in

skip_dupe_check (bool): skip duplicate check for contribution identifiers

Returns:
None

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

any reason to remove the collection type check?
I don't see any other guards around

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Method is defined to only take list[dict]:
def submit_contributions(
self,
contributions: list[dict],
...
)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants