Add support for reading and processing input4MIPs data

[schlunma]

@julienboussard

Description

This PR adds support to read and process input4MIPs data with ESMValCore. If necessary, these data can also be automatically downloaded from ESGF. Since we already support various other ESGF projects, implementing this was rather straightforward.

Unfortunately, it seems that the input4MIPs CMOR tables are not complete and many (maybe most) variables and datasets are not listed in the tables. For example, the controlled vocabulary (CV) lists 28 different dataset (i.e., source_id elements), but the search on ESGF reveals much more (e.g., IAMC-MESSAGE-GLOBIOM-ssp245-1-1 shows in the search, but not in the CV). In addition, most of the variables do not appear in any of the tables (e.g., CO2_em_anthro shows in the search, but cannot be found in any of the tables).

It doesn't seem to be a CMIP7-specific problem because data with mip_era=CMIP6 is also not described in the CV.

Currently, there's no way to read these missing data with ESMValCore.

Code Examples

Recipe interface

# ESMValTool
---
documentation:
  description: Test
  authors:
    - schlund_manuel
  title: Test.

datasets:
  - {project: input4MIPs, mip_era: CMIP6, target_mip: CMIP, dataset: PCMDI-AMIP-1-1-8, grid: gn}

diagnostics:
  test:
    scripts:
      null
    variables:
      tos:
        mip: Omon

Python API:

from ncdata.iris_xarray import cubes_to_xarray
from esmvalcore.dataset import Dataset

dataset = Dataset(
    project="input4MIPs",
    dataset="PCMDI-AMIP-1-1-8",
    mip_era="CMIP6",
    target_mip="CMIP",
    grid="gn",
    short_name="tos",
    mip="Omon",
)

cube = dataset.load()  # this is an iris.cube.Cube
print(cube)

ds = cubes_to_xarray(cube)
print(ds)  # this is an xarray.Dataset

Closes #2770

Link to documentation: TBA

---

Before you get started

• ☝ Create an issue to discuss what you are going to do

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

• 🧪 The new functionality is relevant and scientifically sound
• 🛠 This pull request has a descriptive title and labels
• 🛠 Code is written according to the code quality guidelines
• 🧪 and 🛠 Documentation is available
• 🛠 Unit tests have been added
• 🛠 Changes are backward compatible
• 🛠 Any changed dependencies have been added or removed correctly
• 🛠 The list of authors is up to date
• 🛠 All checks below this pull request were successful

---

To help with the number pull requests:

• 🙏 We kindly ask you to review two other open pull requests in this repository

[codecov]

Codecov Report

Attention: Patch coverage is 94.02985% with 4 lines in your changes missing coverage. Please review.

Project coverage is 95.29%. Comparing base (2cb0787) to head (508452f).

Files with missing lines	Patch %	Lines
esmvalcore/cmor/table.py	94.11%	3 Missing ⚠️
esmvalcore/dataset.py	75.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2774      +/-   ##
==========================================
- Coverage   95.29%   95.29%   -0.01%     
==========================================
  Files         260      260              
  Lines       15357    15396      +39     
==========================================
+ Hits        14635    14671      +36     
- Misses        722      725       +3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[julienboussard]

Hi @schlunma, thank you very much for opening this PR.

As you mentioned, I was able to run the recipe that you created, but was unable to find a way to download the variables I needed (CO2_em_anthro is one of them) because they are not in any tables...

I will let you know if I find a solution, but maybe the best way to do this is to contact the Input4MIPS team so they add them to the tables? Not sure if this is possible.

For now, I think I will download the forcings data directly from the ESGF website.

Thanks again for this, and sorry for not looking into it earlier.