Skip to content

[2178] Add data reader for offgrid inference#2426

Draft
vhertel wants to merge 1 commit into
ecmwf:developfrom
vhertel:vhertel/develop/2178-offgrid-evaluation
Draft

[2178] Add data reader for offgrid inference#2426
vhertel wants to merge 1 commit into
ecmwf:developfrom
vhertel:vhertel/develop/2178-offgrid-evaluation

Conversation

@vhertel
Copy link
Copy Markdown

@vhertel vhertel commented May 29, 2026

Description

Summary

Adds support for offgrid inference, enabling the model to produce forecasts at arbitrary lat/lon target locations rather than being constrained to the original data grid. Introduces a dedicated offgrid data reader, wires it into the multi-stream data sampler, and provides configs for running and evaluating offgrid forecasting.

New config keys

Adds a test_config.offgrid_eval block to the config:

test_config:
  offgrid_eval:
    # absolute path to .npy file with shape (N, 2) [lat, lon] used for offgrid inference
    grid: /e/scratch/weatherai/shared_work/offgrid-test/offgrid_regular.npy
    # temporal spacing between offgrid samples, e.g. 6h
    frequency: 6h
    # TODO add support for geoinfos
    geoinfos: null

How to run

# Train
uv run --offline train --base-config config/config_offgrid_forecasting.yml

# Inference
uv run --offline inference --from-run-id ct961hba --options test_config.output.num_samples=8

# Evaluate
uv run --offline evaluate --config config/evaluate/eval_config_offgrid.yml

Implementation notes

  • In MultiStreamDataSampler, a DataReaderOffgrid is instantiated per file and stream and inherits metadata (channels and normalization) from the original data reader. This lets channels and normalization parameters be loaded from the dataset regardless of type (anemoi, obs, fesom), but the two steps can be merged if preferred.
  • Geoinfos are not yet supported.

Testing

Tested with a uniform grid and with random coordinates for SYNOP. Both grid files are temporarily stored at:

  • /e/scratch/weatherai/shared_work/offgrid-test/offgrid_regular.npy
  • /e/scratch/weatherai/shared_work/offgrid-test/offgrid_synop.npy
map_toqb9dv6_preds_0_2023-10-01T0600_ERA5_global_10v_fstep_001 map_thb1rpuj_preds_0_2023-10-01T0600_ERA5_global_10v_fstep_001

Issue Number

Addresses #2178

Checklist before asking for review

  • I have performed a self-review of my code
  • My changes comply with basic sanity checks:
    • I have fixed formatting issues with ./scripts/actions.sh lint
    • I have run unit tests with ./scripts/actions.sh unit-test
    • I have documented my code and I have updated the docstrings.
    • I have added unit tests, if relevant
  • I have tried my changes with data and code:
    • I have run the integration tests with ./scripts/actions.sh integration-test
    • (bigger changes) I have run a full training and I have written in the comment the run_id(s): launch-slurm.py --time 60
    • (bigger changes and experiments) I have shared a hegdedoc in the github issue with all the configurations and runs for this experiments
  • I have informed and aligned with people impacted by my change:
    • for config changes: the MatterMost channels and/or a design doc
    • for changes of dependencies: the MatterMost software development channel

@vhertel vhertel marked this pull request as draft May 29, 2026 17:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

1 participant