feat(examples): zarr dataset configs and .pdmsh converter for the unified external aero recipe#1828
Draft
negin513 wants to merge 1 commit into
Draft
feat(examples): zarr dataset configs and .pdmsh converter for the unified external aero recipe#1828negin513 wants to merge 1 commit into
negin513 wants to merge 1 commit into
Conversation
… aero recipe Split out of NVIDIA#1798 per review: the drivaer_ml_surface_zarr / drivaer_ml_volume_zarr dataset configs (identical to their .pdmsh twins except the reader: block) and the transitional convert_pdmsh_to_zarr.py CLI. Depends on NVIDIA#1798 for ZarrMeshReader / ZarrDomainMeshReader / save_domain_mesh_to_zarr. Kept out of the flagship recipe until the dataset-migration decision lands. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Split out of #1798 per review discussion (scope proposal): the recipe integration for the mesh-zarr storage path, held as a separate decision from the library core until the team settles whether to migrate curated datasets.
What's included
drivaer_ml_surface_zarr.yaml/drivaer_ml_volume_zarr.yaml: DrivAerML dataset configs readingphysicsnemo-domainmesh-zarrgroups. They differ from their.pdmshtwins only in thereader:block — the full transform stack and all five recipe models run unchanged.src/convert_pdmsh_to_zarr.py: transitional CLI for datasets already curated to.pdmsh(new curation should emit the schema directly from PhysicsNeMo-Curator; companion PR: feat(mesh): DomainMeshZarrSink emitting the physicsnemo mesh-zarr schema physicsnemo-curator#64).dataset_paths.yaml: adds thedrivaer_ml_zarrpath key.Evidence (carried over from #1798, measured with these configs)
.pdmsh) vs 0.1265 (zarr); volume — 0.2648 vs 0.2680 train / 0.0998 vs 0.0987 val at epoch 3; trajectories track epoch-for-epoch within subsample-RNG noise.train.py(volume incl. on-the-fly SDF against the in-group full-resolution STL boundary).benchmark_io, 8 DrivAerML cases, Lustre): 3.66 cold / 7.96 warm batches/s (.pdmsh) vs 9.23 / 23.64 (zarr) — raw logs.Dependencies
Merge-blocks on #1798 (imports
save_domain_mesh_to_zarr/to_cell_soup; configs target${dp:ZarrMeshReader}/${dp:ZarrDomainMeshReader}). Draft until #1798 lands and the dataset-migration decision is made.Checklist
🤖 Generated with Claude Code