Datastructure for FE conditioning - #2706
Conversation
|
@evenmn could you have a look? |
github-actions
Bot
added
data
evenmn
left a comment
evenmn
left a comment
There was a problem hiding this comment.
Looks good to me, just a few comments. The splitting of streams_datasets could perhaps be moved to keep the changes in _init_stream_datasets minimal, but I don't think it's a big issue. I'm looking forward to testing this!
| filenames_cfg = stream_info.get("filenames", [pathlib.Path()]) | ||
| conditioning_cfg = stream_info.get("conditioning", False) | ||
|
|
||
| if conditioning_cfg: |
There was a problem hiding this comment.
Can this logic be moved to DataReaderTimeConditioning, too avoid logics in multi_stream_data_sampler.py that is specific for a data reader?
| conditioning_cfg = stream_info.get("conditioning", False) | ||
|
|
||
| if conditioning_cfg: | ||
| if filenames_cfg is None or len(filenames_cfg) == 0: |
There was a problem hiding this comment.
This handles filenames when it is explicitly set to null/None. However, when the entry is omitted, it defaults to [pathlib.Path()], so this if-statement is not invoked.
Since the default is needed by the processing below, should this condition handle both cases?
if filenames_cfg is None or filenames_cfg == [pathlib.Path()]:2 participants
Description
This PR closes the first subtasks of having forcing in WeatherGen. It included scalar and field data reading and loading.
Run with:
--options streams_directory="./config/streams/era5_1deg_forecasting_conditioning/"Issue Number
Closes #2536 #2537
Checklist before asking for review
./scripts/actions.sh lint./scripts/actions.sh unit-test./scripts/actions.sh integration-testlaunch-slurm.py --time 60