Merge CUAHSI/cssi_evaluation to hydroframe/cssi_evaluation (#17)

* added the nwm snow comparison notebook

* added documentation for snow metrics

* uploadedc the correct documentation files

* removed the wrong document

* added snow.utils module to the evaluation framework. This contains two functions: computing water year, and performing a same-day swe comparison.

* added different day comparison function to snow utils

* formatting

* added snow melt functions to the snow utils module. Split the snow example notebook into a data collection notebook and a single site comparison notebook

* cleaned single site snow comparison notebook by removing unused imports and code

* completed setting up virtual environment on Verde with micromamba, added steps in getting_started.md

* created a new 01 snow notebook for HydroData data retrieval

* changes to package versions for importing packages and snow __init__ for circular import in notebook 02

* added html/css styling in nwm_utils plot_sites_within_domain function for popup formatting

* changed text box formatting, clarified language around 'model outputs'

* remove notebook output

* formatting for comparison_plots() function in nwm_utils.py

* added timeseries/scatter plot figure description text

* Reorganized and added sections for notebook 02; Plotting and description changes for notebook 02 sections 4-5

* cleaned up section 5 of 02 notebook (pre stats section 6 work)

* 02 snow notebook outline for stats section, added multi-site stats summary and plots, condon diagrams

* changed 'relative' bias and 'mean' bias descriptors, fixed the accumulation/ablation compute_stats

* Created a third notebook for the last part of snow comparison (across a watershed)

* add .DS_store and .ipynb_checkpoints to gitignore

* create 01_data_collection_HydroData.ipynb notebook file

* Added getting shapefile from WBD HUC dataset; added Pandas functionality to plot_sites_within_domain() function

* added geojson file to domain_data for CCSS stations. Still have problems using getCCSSData() on Verde

* Added geojson CCSS file for use on Verde. Confirmed that notebook 02 works on Verde up to the 'Summary Metrics at Multiple Sites' section

* wrote loop for accessing CONUS1 SWE with the get_gridded_data function & first plotting

* nearly final Hydrodata notebook; finalized section outline; added hydrodata_utils; removed most of the Cali files

* Final 01 HydroData SWE Notebook cleanup, update mapping function in nwm_utils, readded domain shapefiles

* removed old commented-out mapping function

* deleted old 01 Hydrodata notebook

* Revert "Cam 965 test snow notebooks and revise snow 02 notebook"

* replaced the DonPedroDam shapefiles that were accidentally removed

* renamed 02 snow notebook

* PF evaluation for snow 02 notebook complete, EXCEPT for stats section

* changed some NWM wording in a markdown box

* create new 03 snow notebook for ParFlow/Hydrodata

* added specific model and output folders for PF outputs

* added coordinate grab for PF-CONUS grid cells to 01, 03 modifications

* nearly complete ParFlow snow notebook 03; some modifications for NWM snow notebook 03; some package and function updates

* added nwm example

* added evaluation to nwm notebook

* moved files to new directories

* Created new folders for reorganizing files

* Added all necessary python files, init files; Added a General README

* added header descriptions in empty files

* Migrated most functions to their appropriate files for reorg. Not including model specific and some observational functions.

* Checked package imports for all new files, except model specific ones

* created 'collect_observations' directory in /examples and moved the original 01_data_collection.ipynb notebooks. Renamed these for PF/HydroData and NWM

* edited files for missing imports requirements

* Modify dataCollectionHydrodata_parflow.ipynb for module reorg. Can confirm notebook works.

* Modify dataCollection_nwm.ipynb for module reorg. Can confirm notebook works.

* final changes before PR

* Moved the root-level readme file to the docs folder for now. Some of the content is specific to Parflow. We can re-use the content of this file later when the reorganization of the github repository is finalized.

* Minor updates to the readme file.

* Add a readme to describe both the workflow and the entire repository

* Add an initial workflow chart for the hackathon activities

* Update the environment setup flow to include a one-time setup step that supports both the core library configuration for the evaluation workflow/package and the runtime dependencies.

* updated instruction to install deps

* tested running notebooks with the new env set-up, data collection works fine. NWM and Parflow notebooks (#2) should be updated. They still point to cssi_evaluation.snow which no longer exists in our reorg branch

* removed backup files b/c the updated env set-up workflow works fine

* revised deps based on Amy's feedback

---------

Co-authored-by: Anthony Castronova <castronova.anthony@gmail.com>
Co-authored-by: danielletijerina <dtt2@princeton.edu>
Co-authored-by: Danielle Tijerina-Kreuzer <danielletk@Danielles-CUAHSI-MacBook.local>
Co-authored-by: danielletijerina <dtijerina@cuahsi.org>
Co-authored-by: Irene Garousi-Nejad <igarousi@Irene-CUAHSI.local>

Author: 5 people

README.md

@@ -1,38 +1,176 @@
-# cssi_evaluation
-
-Code used to compare ParFlow simulated output to real-world observations.
-
-Please see `example_workflow.ipynb` for an example of how the functions in this module are intended to be used with each other. Note that the input in this example is a mask generated from a HUC (list) using `subsettools`. This is one method of generating a mask, but the workflow will work with any mask and accompanying bounds within either the conus1 or conus2 domain. This workflow is restricted to comparisons within the conus1 or conus2 domains. If you wish to use the entire conus1 or conus2 domain, set `ij_bounds = None` and use `utils.get_conus_mask("conus2")` to obtain the mask for the full conus2 grid (or likewise for conus1).
-
-This module contains three distinct steps, which will be linked up in the final workflow:
-  1. Gather site-level observations for a requested domain
-  2. Extract and format output from ParFlow grid cells that match up with site locations
-  3. Calculate metrics and produce plots to compare outputs from (1) and (2)
-
-Supported variables:
-  - 'streamflow' (USGS) ('hourly' or 'daily')
-  - 'water_table_depth' (USGS) ('hourly' or 'daily')
-  - 'swe' (SNOTEL) ('daily')
-  - 'latent_heat' (AmeriFlux) ('hourly') 
-
-Supported metrics:
-  - 'r2': Correlation of determination
-  - 'spearman_rho': Spearman's rank correlation coefficient
-  - 'mse': Mean Squared Error
-  - 'rmse': Root Mean Squared Error
-  - 'bias': bias
-  - 'percent_bias': percent bias
-  - 'abs_rel_bias': absolute relative bias
-  - 'total_difference': total difference (ParFlow minus observations)
-  - 'pearson_r': Pearson's R
-  - 'nse': Nash-Sutcliffe Efficiency
-  - 'kge': Kling-Gupta Efficiency
-  - 'bias_from_r': bias from R ([equation 16](https://www.nature.com/articles/srep19401))
-  - 'condon': Condon category (low/high bias, poor/good shape)
-
-Supported plots (see `plots.py` for full API details):
-  - `plot_obs_locations()`: Given observation metadata, plot site locations within a mask. Sites are color-coded by site type, if multiple types of sites are present.
-  - `plot_time_series()`: Plot ParFlow time series against observation time series; one plot per site.
-  - `plot_compare_scatter()`: Plot a single scatterplot comparing the average values for all sites for ParFlow vs. observations.
-  - `plot_metric_map()`: Plot sites within a mask, colored by their value on a given comparison metric; one plot per metric.
-  - `plot_condon_diagram()`: Plot Condon diagram comparing absolute relative bias to Spearman's rho.
+# Hydrologic Model Evaluation Framework
+
+This repository contains a hydrologic model evaluation framework for comparing modeled outputs against observations and reference datasets at national scale. The goal is not only to help users find data, but to provide a structured, reproducible workflow for model evaluation and benchmarking across hydrologic models, variables, and spatial scales.
+
+This effort builds on earlier NSF-supported work from the HydroGEN and HydroFrame projects. Those projects focused on building and serving large-scale hydrologic modeling capabilities. In that work, it became clear that community-scale options for evaluating national and regional model outputs remain limited. `HydroData` was initially developed to meet internal project needs for data access, but broader engagement showed a clear need for a framework that connects those data resources to evaluation workflows. This repository, aka `cssi_evaluation`, addresses that need.
+
+## What this framework does
+
+The framework brings together several pieces that operate as one workflow:
+
+1. Model-specific adapters that extract and **format model outputs**.
+2. Data-access tools that **retrieve observations** and reference datasets.
+3. Shared evaluation utilities for **preprocessing, statistics, and plotting**.
+4. **Variable-specific diagnostics** for process-aware evaluation of particular hydrologic variables.
+
+The operational framework lives under [`src/cssi_evaluation/`](src/cssi_evaluation). The top-level [`docs/`](docs), [`examples/`](examples), and [`tests/`](tests) directories support the framework, but they are not themselves workflow stages.
+
+## Current scope
+
+We have started developing this framework as a **Python** package to facilitate evaluation of modeled results against real-world data. At present, the repository reflects two main modeling paths:
+
+- ParFlow-oriented evaluation, especially for ParFlow-CONUS2.1
+- National Water Model (NWM) workflows that are being organized alongside the same framework structure
+
+Future development is expected to add both additional site-based datasets and gridded remote-sensing datasets.
+
+HydroData-related packages such as `hf_hydrodata` and `subsettools` are external dependencies used by this framework. They are not part of the source tree in this repository, but they are important data-access inputs, especially for the ParFlow-oriented workflow. Using those packages allows acquisition of comparison datasets to remain reproducible in code.
+
+## How users interact with the framework
+
+Users primarily interact with the package through Python functions and example notebooks. Typical workflows allow a user to define:
+
+- set up environment with required packages
+- define a domain of interest by HUC, latitude/longitude bounding box, or upstream drainage area
+- define a time range for evaluation
+- select one or more observational variables to compare against model output
+
+The package includes shared statistical metrics such as RMSE, MSE, Pearson correlation, Spearman rank correlation, Nash-Sutcliffe Efficiency, Kling-Gupta Efficiency, R-squared, bias, percent bias, absolute relative bias, total difference, and Condon category. It also includes plotting utilities for site-level time series and mapped summaries of evaluation metrics across sites.
+
+## General metrics and variable-specific diagnostics
+
+The repository intentionally separates model-agnostic evaluation tools from diagnostics that are specific to a variable or process.
+
+- General metrics apply broadly across time series and model types.
+- Variable-specific diagnostics are evaluation methods designed around the scientific behavior of a particular hydrologic variable.
+
+For example, continuous time-series metrics are useful but may not fully capture seasonal snow behavior. Snow evaluation often requires targeted diagnostics such as:
+
+- peak SWE same-day comparison
+- peak SWE different-day comparison
+- melt timing comparison
+
+These diagnostics complement general statistical metrics by focusing on process-relevant behaviors rather than only full-series agreement.
+
+## Framework workflow
+
+The framework standardizes modeled and observational data before applying shared utilities and variable-specific diagnostic functions. 
+Dashed arrows indicate repository locations, not workflow direction. Blue boxes represent core workflow components, green circles show 
+outputs generated by the workflow, and yellow boxes denote workflow sections.
+
+
+```mermaid
+flowchart LR
+    subgraph A[Model Output Sources]
+        direction LR
+        A1[ParFlow outputs]
+        A2[NWM outputs]
+        A3[Future model<br/>outputs]
+    end
+
+    subgraph B[Observation Data Sources]
+        direction LR
+        BS[" "]
+        B1[HydroData-related<br/>packages]
+        B2[External observation<br/>sources]
+        B3[Other reference<br/>datasets]
+    end
+
+    E([Model-Specific Adapters])
+    F([Data Access Layer])
+    I((Standardized evaluation-ready data))
+
+    subgraph C[General Evaluation Utilities]
+        C1[General statistical metrics]
+        C2[Plots and summaries]
+    end
+
+    subgraph D[Process-aware Evaluation Utilities]
+        direction LR
+        DS[" "]
+        D1[Streamflow diagnostics]
+        D2[Snow diagnostics]
+        D3[Additional variables]
+    end
+
+    G([Shared Evaluation Utils])
+    H([Variable-Specific Diagnostics])
+
+    A --> E
+    B --> F
+    E --> I
+    F --> I
+    I --> C
+    I --> D
+    C --> G
+    D --> H
+    G --> J((Evaluation Results))
+    H --> J
+
+    subgraph P[Repository Paths]
+        direction LR
+        P1[src/cssi_evaluation/models/]
+        P2[src/cssi_evaluation/external_data_access/]
+        P3[src/cssi_evaluation/utils/]
+        P4[src/cssi_evaluation/variables/]
+    end
+
+    E -.-> P1
+    F -.-> P2
+    G -.-> P3
+    H -.-> P4
+
+    classDef sourcegroup fill:#d9d9d9,stroke:#7a7a7a,color:#111;
+    classDef source fill:#efe8db,stroke:#8a6b2f,color:#111;
+    classDef framework fill:#bcdcf5,stroke:#1f5f99,color:#111;
+    classDef results fill:#dff5df,stroke:#3c8c3c,color:#111;
+    classDef repo fill:#7a7a7a,stroke:#4d4d4d,color:#111;
+    classDef path fill:#b3b3b3,stroke:#4d4d4d,color:#111;
+    classDef spacer fill:transparent,stroke:transparent,color:transparent;
+
+    class A,B,C,D sourcegroup;
+    class A1,A2,A3,B1,B2,B3,C1,C2,C3,D1,D2,D3 source;
+    class E,F,H,G framework;
+    class I,J results;
+    class BS,DS spacer;
+    class E1,F1,Z,Y path;
+    linkStyle 10,11,12,13 stroke:#333333,stroke-width:2px,stroke-dasharray:6 4;
+```
+
+## Repository structure
+
+```text
+cssi_evaluation/
+├── src/cssi_evaluation/         # Core framework code
+│   ├── models/                  # Model-specific adapters
+│   ├── external_data_access/    # Observation and reference-data access helpers
+│   ├── utils/                   # Shared evaluation utilities
+│   ├── variables/               # Variable-specific diagnostics
+│   └── example_workflow.ipynb   # Package-level example workflow
+├── examples/                    # Example notebooks and supporting assets
+├── docs/                        # Project documentation and notes
+├── tests/                       # Package tests
+├── pyproject.toml               # Package metadata and dependencies
+└── README.md
+```
+
+## What is and is not part of the workflow
+
+The framework logic is the code under [`src/cssi_evaluation/`](src/cssi_evaluation).
+
+The following directories are important, but they are not part of the workflow implementation itself:
+
+- [`docs/`](docs) for narrative documentation and project references
+- [`examples/`](examples) for demonstration notebooks and usage examples
+- [`tests/`](tests) for validation and regression testing
+
+## Outlook
+
+The framework is intended to grow by adding:
+
+- new model adapters
+- new observation and reference-data pathways
+- new variable-specific diagnostics
+- clearer workflows for users bringing their own model outputs, including both physics-based and ML-based models
+
+The long-term aim is to reduce the barrier to reproducible hydrologic model evaluation while keeping the code structure aligned with the scientific workflow.

cssi_env.yml

@@ -0,0 +1,21 @@
+name: cssi_evaluation
+channels:
+  - conda-forge
+dependencies:
+  - python=3.10
+  - pip
+  - gdal
+  - geopandas=0.12.2
+  - rasterio=1.3.6
+  - fiona
+  - rioxarray=0.15.1
+  - pyproj
+  - libtiff=4.5.0
+  - holoviews=1.19.0
+  - geoviews=1.11.0
+  - param=2.0.1
+  - nodejs
+  - zarr=2.13.3
+  - s3fs=2023.6.0
+  - pip:
+      - "-e .[notebooks]"

docs/2026_modeling_hackathon.md

@@ -0,0 +1,55 @@
+This workflow chart outlines the structure of the 2026 Modeling Hackathon. It shows how participants move from environment setup into the ParFlow and National Water Model (NWM) evaluation paths, and highlights how additional model workflows can be integrated into the same framework in future hackathons.
+
+```mermaid
+flowchart TD
+    A[Participant enters workshop] --> B[Set up environment]
+    B --> B1[HydroData login]
+    B --> B2[GitHub access]
+    B --> B3[Python environment]
+    B --> B4[Test imports or Jupyter example]
+
+    B4 --> C{Which model path?}
+
+    C --> D[Day 1: ParFlow path]
+    C --> E[Day 2: NWM path]
+    C --> F[Future model path]
+
+    D --> D1[Access observations through HydroData]
+    D1 --> D2[Read ParFlow outputs with model-specific tools]
+    D2 --> D3[Apply shared utilities]
+    D3 --> D4[Run snow or streamflow evaluation]
+    D4 --> D5[Review plots and metrics]
+
+    E --> E1[Access reference data from external links]
+    E1 --> E2[Read NWM outputs with model-specific tools]
+    E2 --> E3[Apply shared utilities]
+    E3 --> E4[Run variable-specific evaluation]
+    E4 --> E5[Review plots and metrics]
+
+    F --> F1[Create a new model-specific adapter]
+    F1 --> F2[Match model outputs to framework format]
+    F2 --> F3[Connect observations or reference data]
+    F3 --> F4[Reuse shared metrics and plotting]
+    F4 --> F5[Add variable-specific methods as needed]
+
+    D5 --> G[Collect feedback]
+    E5 --> G
+    F5 --> G
+
+    G --> G1[Challenges and incompatibilities]
+    G --> G2[Integration opportunities]
+    G --> G3[Needed data sources]
+    G --> G4[Needed variables and metrics]
+
+    classDef start fill:#d9edf7,stroke:#31708f,color:#111;
+    classDef day1 fill:#e7f4e4,stroke:#3d7a3a,color:#111;
+    classDef day2 fill:#fff1cc,stroke:#9b7a19,color:#111;
+    classDef future fill:#f4e1f5,stroke:#8d4a8f,color:#111;
+    classDef feedback fill:#f9e0e6,stroke:#9c3758,color:#111;
+
+    class A,B,B1,B2,B3,B4,C start;
+    class D,D1,D2,D3,D4,D5 day1;
+    class E,E1,E2,E3,E4,E5 day2;
+    class F,F1,F2,F3,F4,F5 future;
+    class G,G1,G2,G3,G4 feedback;
+```

docs/Parflow-related-readme.md

@@ -0,0 +1,38 @@
+# cssi_evaluation
+
+Code used to compare ParFlow simulated output to real-world observations.
+
+Please see `example_workflow.ipynb` for an example of how the functions in this module are intended to be used with each other. Note that the input in this example is a mask generated from a HUC (list) using `subsettools`. This is one method of generating a mask, but the workflow will work with any mask and accompanying bounds within either the conus1 or conus2 domain. This workflow is restricted to comparisons within the conus1 or conus2 domains. If you wish to use the entire conus1 or conus2 domain, set `ij_bounds = None` and use `utils.get_conus_mask("conus2")` to obtain the mask for the full conus2 grid (or likewise for conus1).
+
+This module contains three distinct steps, which will be linked up in the final workflow:
+  1. Gather site-level observations for a requested domain
+  2. Extract and format output from ParFlow grid cells that match up with site locations
+  3. Calculate metrics and produce plots to compare outputs from (1) and (2)
+
+Supported variables:
+  - 'streamflow' (USGS) ('hourly' or 'daily')
+  - 'water_table_depth' (USGS) ('hourly' or 'daily')
+  - 'swe' (SNOTEL) ('daily')
+  - 'latent_heat' (AmeriFlux) ('hourly') 
+
+Supported metrics:
+  - 'r2': Correlation of determination
+  - 'spearman_rho': Spearman's rank correlation coefficient
+  - 'mse': Mean Squared Error
+  - 'rmse': Root Mean Squared Error
+  - 'bias': bias
+  - 'percent_bias': percent bias
+  - 'abs_rel_bias': absolute relative bias
+  - 'total_difference': total difference (ParFlow minus observations)
+  - 'pearson_r': Pearson's R
+  - 'nse': Nash-Sutcliffe Efficiency
+  - 'kge': Kling-Gupta Efficiency
+  - 'bias_from_r': bias from R ([equation 16](https://www.nature.com/articles/srep19401))
+  - 'condon': Condon category (low/high bias, poor/good shape)
+
+Supported plots (see `plots.py` for full API details):
+  - `plot_obs_locations()`: Given observation metadata, plot site locations within a mask. Sites are color-coded by site type, if multiple types of sites are present.
+  - `plot_time_series()`: Plot ParFlow time series against observation time series; one plot per site.
+  - `plot_compare_scatter()`: Plot a single scatterplot comparing the average values for all sites for ParFlow vs. observations.
+  - `plot_metric_map()`: Plot sites within a mask, colored by their value on a given comparison metric; one plot per metric.
+  - `plot_condon_diagram()`: Plot Condon diagram comparing absolute relative bias to Spearman's rho.