Add Zero Emission Commitment metric by bouweandela · Pull Request #204 · Climate-REF/climate-ref

bouweandela · 2025-03-26T11:42:35Z

Description

Add Zero Emission Commitment metric. Example data added in Climate-REF/ref-sample-data#21.

Usage

Because the diagnostic script is not yet available in a released version of ESMValTool, a development installation is needed to run this metric. This can be achieved with the following steps, where I have used software = "/home/bandela/climate_ref/software" in ref.toml:

Create the virtual conda environment for ESMValTool ref providers create-env --provider esmvaltool. This will list the path to the environment, e.g. /home/bandela/climate_ref/software/conda/esmvaltool-e5744788ae141d3dd500d239ce6d8fecceffaa1. Use this path in the next step.
Install ESMValTool from source in the environment: /home/bandela/climate_ref/software/conda/micromamba run -p /home/bandela/climate_ref/software/conda/esmvaltool-e5744788ae141d3dd500d239ce6d8fecceffaa19 pip install --no-deps git+https://github.com/ESMValGroup/ESMValTool.git

Checklist

Please confirm that this pull request has done the following:

Tests added
Documentation added (where applicable)
Changelog item added to changelog/

Closes #194

codecov · 2025-03-26T11:44:55Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Files with missing lines	Coverage Δ
...rc/cmip_ref_metrics_esmvaltool/metrics/__init__.py	`100.00% <100.00%> (ø)`
...ool/src/cmip_ref_metrics_esmvaltool/metrics/zec.py	`100.00% <100.00%> (ø)`
...mvaltool/src/cmip_ref_metrics_esmvaltool/recipe.py	`92.85% <ø> (ø)`

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

lewisjared · 2025-03-31T21:50:00Z

Sample data being added in #220

lewisjared

I get the following failure when running this locally. I'm assuming that will fail until a new ESMValTool release is available on conda so we can bump the environment.yaml file

INFO    [16660] 
______________________________________________________________________
          _____ ____  __  ____     __    _ _____           _
         | ____/ ___||  \/  \ \   / /_ _| |_   _|__   ___ | |
         |  _| \___ \| |\/| |\ \ / / _` | | | |/ _ \ / _ \| |
         | |___ ___) | |  | | \ V / (_| | | | | (_) | (_) | |
         |_____|____/|_|  |_|  \_/ \__,_|_| |_|\___/ \___/|_|
______________________________________________________________________

Earth System Model Evaluation Tool

A community tool for the evaluation of Earth system models.

https://esmvaltool.org

The Earth System Model Evaluation Tool (ESMValTool) is a community
diagnostics and performance metrics tool for the evaluation of Earth
System Models (ESMs) that allows for routine comparison of single or
multiple models, either against predecessor versions or against
observations.

Tutorial: https://tutorial.esmvaltool.org
Documentation: https://docs.esmvaltool.org
Contact: esmvaltool-dev@listserv.dfn.de

If you find this software useful for your research, please cite it using
https://doi.org/10.5281/zenodo.3387139 for ESMValCore or
https://doi.org/10.5281/zenodo.3401363 for ESMValTool or
any of the reference papers listed at https://esmvaltool.org/references/.

Have fun!

INFO    [16660] Package versions
INFO    [16660] ----------------
INFO    [16660] ESMValCore: 2.12.0
INFO    [16660] ESMValTool: 2.12.0
INFO    [16660] ----------------
INFO    [16660] Reading configuration files from:
/Users/jared/code/CMIP-REF/cmip-ref/.ref/software/conda/esmvaltool-e5744788ae141d3dd500d239ce6d8fecceffaa19/lib/python3.12/site-packages/esmvalcore/config/configurations/defaults (defaults)
/Users/jared/.config/esmvaltool [NOT AN EXISTING DIRECTORY] (default user configuration directory)
/Users/jared/code/CMIP-REF/cmip-ref/.ref/scratch/esmvaltool/esmvaltool-zero-emission-commitment/7cf8c771d0eb3171cef4cdda69af72e3499e0c98/config (command line argument)
INFO    [16660] Writing program log files to:
/Users/jared/code/CMIP-REF/cmip-ref/.ref/scratch/esmvaltool/esmvaltool-zero-emission-commitment/7cf8c771d0eb3171cef4cdda69af72e3499e0c98/results/recipe_20250407_172808/run/main_log.txt
/Users/jared/code/CMIP-REF/cmip-ref/.ref/scratch/esmvaltool/esmvaltool-zero-emission-commitment/7cf8c771d0eb3171cef4cdda69af72e3499e0c98/results/recipe_20250407_172808/run/main_log_debug.txt
/Users/jared/code/CMIP-REF/cmip-ref/.ref/scratch/esmvaltool/esmvaltool-zero-emission-commitment/7cf8c771d0eb3171cef4cdda69af72e3499e0c98/results/recipe_20250407_172808/run/cmor_log.txt
WARNING [16660] /Users/jared/code/CMIP-REF/cmip-ref/.ref/software/conda/esmvaltool-e5744788ae141d3dd500d239ce6d8fecceffaa19/lib/python3.12/site-packages/esmpy/interface/loadESMF.py:94: VersionWarning: ESMF installation version 8.8.0, ESMPy version 8.8.0b0
  warnings.warn("ESMF installation version {}, ESMPy version {}".format(

INFO    [16660] Starting the Earth System Model Evaluation Tool at time: 2025-04-07 17:28:08 UTC
INFO    [16660] ----------------------------------------------------------------------
INFO    [16660] RECIPE   = /Users/jared/code/CMIP-REF/cmip-ref/.ref/scratch/esmvaltool/esmvaltool-zero-emission-commitment/7cf8c771d0eb3171cef4cdda69af72e3499e0c98/recipe.yml
INFO    [16660] RUNDIR     = /Users/jared/code/CMIP-REF/cmip-ref/.ref/scratch/esmvaltool/esmvaltool-zero-emission-commitment/7cf8c771d0eb3171cef4cdda69af72e3499e0c98/results/recipe_20250407_172808/run
INFO    [16660] WORKDIR    = /Users/jared/code/CMIP-REF/cmip-ref/.ref/scratch/esmvaltool/esmvaltool-zero-emission-commitment/7cf8c771d0eb3171cef4cdda69af72e3499e0c98/results/recipe_20250407_172808/work
INFO    [16660] PREPROCDIR = /Users/jared/code/CMIP-REF/cmip-ref/.ref/scratch/esmvaltool/esmvaltool-zero-emission-commitment/7cf8c771d0eb3171cef4cdda69af72e3499e0c98/results/recipe_20250407_172808/preproc
INFO    [16660] PLOTDIR    = /Users/jared/code/CMIP-REF/cmip-ref/.ref/scratch/esmvaltool/esmvaltool-zero-emission-commitment/7cf8c771d0eb3171cef4cdda69af72e3499e0c98/results/recipe_20250407_172808/plots
INFO    [16660] ----------------------------------------------------------------------
INFO    [16660] Running tasks using at most 10 processes
INFO    [16660] If your system hangs during execution, it may not have enough memory for keeping this number of tasks in memory.
INFO    [16660] If you experience memory problems, try reducing 'max_parallel_tasks' in your configuration.
INFO    [16660] Creating tasks from recipe
INFO    [16660] Creating tasks for diagnostic zec
INFO    [16660] Creating diagnostic task zec/zec
INFO    [16660] Maximum memory used (estimate): 0.0 GB
INFO    [16660] Sampled every second. It may be inaccurate if short but high spikes in memory consumption occur.
ERROR   [16660] Program terminated abnormally, see stack trace below for more information:
Traceback (most recent call last):
  File "/Users/jared/code/CMIP-REF/cmip-ref/.ref/software/conda/esmvaltool-e5744788ae141d3dd500d239ce6d8fecceffaa19/lib/python3.12/site-packages/esmvalcore/_main.py", line 603, in run
    fire.Fire(ESMValTool())
  File "/Users/jared/code/CMIP-REF/cmip-ref/.ref/software/conda/esmvaltool-e5744788ae141d3dd500d239ce6d8fecceffaa19/lib/python3.12/site-packages/fire/core.py", line 135, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jared/code/CMIP-REF/cmip-ref/.ref/software/conda/esmvaltool-e5744788ae141d3dd500d239ce6d8fecceffaa19/lib/python3.12/site-packages/fire/core.py", line 468, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
                                ^^^^^^^^^^^^^^^^^^^^
  File "/Users/jared/code/CMIP-REF/cmip-ref/.ref/software/conda/esmvaltool-e5744788ae141d3dd500d239ce6d8fecceffaa19/lib/python3.12/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jared/code/CMIP-REF/cmip-ref/.ref/software/conda/esmvaltool-e5744788ae141d3dd500d239ce6d8fecceffaa19/lib/python3.12/site-packages/esmvalcore/_main.py", line 429, in run
    self._run(recipe, session, cli_config_dir)
  File "/Users/jared/code/CMIP-REF/cmip-ref/.ref/software/conda/esmvaltool-e5744788ae141d3dd500d239ce6d8fecceffaa19/lib/python3.12/site-packages/esmvalcore/_main.py", line 486, in _run
    process_recipe(recipe_file=recipe, session=session)
  File "/Users/jared/code/CMIP-REF/cmip-ref/.ref/software/conda/esmvaltool-e5744788ae141d3dd500d239ce6d8fecceffaa19/lib/python3.12/site-packages/esmvalcore/_main.py", line 132, in process_recipe
    recipe = read_recipe_file(recipe_file, session)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jared/code/CMIP-REF/cmip-ref/.ref/software/conda/esmvaltool-e5744788ae141d3dd500d239ce6d8fecceffaa19/lib/python3.12/site-packages/esmvalcore/_recipe/recipe.py", line 76, in read_recipe_file
    return Recipe(raw_recipe, session, recipe_file=filename)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jared/code/CMIP-REF/cmip-ref/.ref/software/conda/esmvaltool-e5744788ae141d3dd500d239ce6d8fecceffaa19/lib/python3.12/site-packages/esmvalcore/_recipe/recipe.py", line 817, in __init__
    self.tasks = self.initialize_tasks()
                 ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jared/code/CMIP-REF/cmip-ref/.ref/software/conda/esmvaltool-e5744788ae141d3dd500d239ce6d8fecceffaa19/lib/python3.12/site-packages/esmvalcore/_recipe/recipe.py", line 1169, in initialize_tasks
    tasks = self._create_tasks()
            ^^^^^^^^^^^^^^^^^^^^
  File "/Users/jared/code/CMIP-REF/cmip-ref/.ref/software/conda/esmvaltool-e5744788ae141d3dd500d239ce6d8fecceffaa19/lib/python3.12/site-packages/esmvalcore/_recipe/recipe.py", line 1131, in _create_tasks
    new_tasks = self._create_diagnostic_tasks(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jared/code/CMIP-REF/cmip-ref/.ref/software/conda/esmvaltool-e5744788ae141d3dd500d239ce6d8fecceffaa19/lib/python3.12/site-packages/esmvalcore/_recipe/recipe.py", line 1042, in _create_diagnostic_tasks
    task = DiagnosticTask(
           ^^^^^^^^^^^^^^^
  File "/Users/jared/code/CMIP-REF/cmip-ref/.ref/software/conda/esmvaltool-e5744788ae141d3dd500d239ce6d8fecceffaa19/lib/python3.12/site-packages/esmvalcore/_task.py", line 383, in __init__
    self.cmd = self._initialize_cmd()
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jared/code/CMIP-REF/cmip-ref/.ref/software/conda/esmvaltool-e5744788ae141d3dd500d239ce6d8fecceffaa19/lib/python3.12/site-packages/esmvalcore/_task.py", line 411, in _initialize_cmd
    raise DiagnosticError(f"{err_msg}: file does not exist.")
esmvalcore._task.DiagnosticError: Cannot execute script 'climate_metrics/zec.py' (/Users/jared/code/CMIP-REF/cmip-ref/climate_metrics/zec.py): file does not exist.
INFO    [16660] 
If you have a question or need help, please start a new discussion on https://github.com/ESMValGroup/ESMValTool/discussions
If you suspect this is a bug, please open an issue on https://github.com/ESMValGroup/ESMValTool/issues
To make it easier to find out what the problem is, please consider attaching the files run/recipe_*.yml and run/main_log_debug.txt from the output directory.

…tric-esmvaltool-zec

bouweandela · 2025-04-08T13:03:07Z

Thanks for reviewing @lewisjared! I haven't been able to install ESMValTool from source through the lock file, so for now I would recommend just installing the development version of ESMValtool into the ESMValTool conda environment. I added instructions on how to do that in the top post.

lewisjared · 2025-04-08T13:08:50Z

I'll give that a go. Are you happy to wait until a new ESMValTool release is available before merging? Otherwise, we will need to guard against this expected failure in the integration tests or add the install from main step to the CI.

bouweandela · 2025-04-08T15:08:57Z

Are you happy to wait until a new ESMValTool release is available before merging?

There is no release date yet for the next ESMValTool release, so I would prefer not to wait for it.

bouweandela · 2025-04-08T15:58:37Z

I'll see if I can add an extra option to the ref providers create-env command to install the development version.

bouweandela · 2025-04-09T09:09:48Z

Done in #236

mikapfl

Nice, works for me with the example data and dev version of ESMValTool. If we can get the dev version automatically via the provider, looks good to me.

bouweandela marked this pull request as ready for review March 26, 2025 14:28

bouweandela mentioned this pull request Mar 26, 2025

Add ZEC sample data Climate-REF/ref-sample-data#21

Merged

3 tasks

Add Zero Emission Commitment metric

8ef818f

bouweandela force-pushed the add-metric-esmvaltool-zec branch from 7e33d50 to 8ef818f Compare March 26, 2025 20:46

bouweandela requested a review from mikapfl March 27, 2025 13:12

lewisjared reviewed Apr 7, 2025

View reviewed changes

Merge branch 'main' of github.com:Climate-REF/climate-ref into add-me…

1293524

…tric-esmvaltool-zec

mikapfl approved these changes Apr 9, 2025

View reviewed changes

bouweandela merged commit ae95196 into Climate-REF:main Apr 9, 2025
12 checks passed

bouweandela deleted the add-metric-esmvaltool-zec branch April 9, 2025 13:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Zero Emission Commitment metric#204

Add Zero Emission Commitment metric#204
bouweandela merged 2 commits intoClimate-REF:mainfrom
bouweandela:add-metric-esmvaltool-zec

bouweandela commented Mar 26, 2025 •

edited

Loading

Uh oh!

codecov bot commented Mar 26, 2025 •

edited

Loading

Uh oh!

lewisjared commented Mar 31, 2025

Uh oh!

lewisjared left a comment

Uh oh!

bouweandela commented Apr 8, 2025

Uh oh!

lewisjared commented Apr 8, 2025

Uh oh!

bouweandela commented Apr 8, 2025

Uh oh!

bouweandela commented Apr 8, 2025

Uh oh!

bouweandela commented Apr 9, 2025

Uh oh!

mikapfl left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

bouweandela commented Mar 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Usage

Checklist

Uh oh!

codecov bot commented Mar 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

lewisjared commented Mar 31, 2025

Uh oh!

lewisjared left a comment

Choose a reason for hiding this comment

Uh oh!

bouweandela commented Apr 8, 2025

Uh oh!

lewisjared commented Apr 8, 2025

Uh oh!

bouweandela commented Apr 8, 2025

Uh oh!

bouweandela commented Apr 8, 2025

Uh oh!

bouweandela commented Apr 9, 2025

Uh oh!

mikapfl left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

bouweandela commented Mar 26, 2025 •

edited

Loading

codecov bot commented Mar 26, 2025 •

edited

Loading