ctsm5.4.000: Update to CMIP7 LULCC forcings

[wwieder]

It looks like a preliminary dataset for LULCC is available for CMIP7 (scroll down the table with links)

Specifically, is the v3.0 dataset any different that what we're already using from TRENDY?
This github page could be helpful for tracking changes / updates to the dataset.

If this dataset is different from TRENDY, we should migrate to the latest v3 data so we're testing the model with LULCC data that's hopefully consistent with the final CMIP7 data.

@lawrencepj1 can you have a look at this dataset to see if it's different that what we've already integrated into the 5.3 tag?

Definition of Done:

• @ekluzek create the alpha-ctsm5.4.CMIP7 branch and get it setup
• @lawrencepj1 processes the new CMIP7 data
• @lawrencepj1 does his verification of the new raw datasets
• @olyson processes urban with Peter's new raw datasets
• @wwieder creates f09, ne30 datasets with above
• Verify the f09 and ne30 datasets
• @olyson Run simulations with the new datasets at f09 and ne30
• Verify based on the output that things are reasonable @olyson @wwieder
• Move new raw data to /inputdata (their permanent location) (SSP raw data not available, yet)
• Update to CMIP7 stream files as per New population dataset for streams files #2701 now replaced by ctsm5.3.082: Update to official CMIP7 population density data #3545
• @slevis-lmwg opens PR to add the new datasets into the XML in the alpha branch (alpha-ctsm5.4.CMIP7)
• Finalize the Grids in CESM3 document
• Add support for new grids: ne3pg2, mpasa30, ne120pg3-landuse timeseries, ne0x8_NATL in Makefile
• Testing of mksurfdata_esmf with the new datasets (make all worked for all the ctsm53 grids)
• Verify that everything needed is built in the Makefile (see Logistical improvements to fsurdat/landuse files #2337)
• For SSP's point to the CMIP6 1850-SSP surface dataset to enable future scenarios for CMIP6 with the previous datasets (see ISSP tests need to point to CMIP6 ctsm5.3 fsurdat/landuse files to PASS #3250)
• Run SSP test cases for f09 for CMIP6 work with the changes to handle CMIP6/CMIP7 (successfully tested in alpha-ctsm5.4.CMIP7.12.ctsm5.3.075: Add support for CLM_CMIP_ERA (= "cmip6" or "cmip7") #3501)
• Make sure the tasks needed in the ctsm5.4 project are completed
• Make sure the tasks in the ctsm5.4 milestone are completed that need to be before dataset creation; if we have new NOANTHRO raw data, remove comment in cime_config/config_component.xml (and elsewhere?) that we haven't updated this raw dataset
• Create new datasets for all resolutions: except the subset data ones! (Originally alpha-ctsm5.4.CMIP7.13.ctsm5.3.079: Prepare Makefile to make CTSM5.4 fsurdat/landuse datasets #3482 and currently alpha-ctsm5.4.CMIP7.14.ctsm5.3.079: New raw lai, soilcolor, 1700-2023 landuse files, and NOANTHRO landuse file #3555 and tag alpha-ctsm5.4.CMIP7.14.ctsm5.3.079)
• update tools/site_and_regional/default_data_1850.cfg and default_data_2000.cfg files to be consistent with default 5.4 surface datasets
• Run make all-subset (PR alpha-ctsm5.4.CMIP7.15.ctsm5.3.079: Updates to default_data_[1850,2000].cfg & modify_smallville.sh #3562)
• Was it accidental or intentional that we didn't have a checkbox about making NEON/PLUMBER datasets? In any case DONE.
• Update namelist_defaults for the new fsurdat/landuse paths. Included this correction. Questions and steps:
  • OK ./run_sys_tests -s ctsm_sci --skip-compare --skip-generate and build-namelist_test.pl
  • Erik, keep or remove ctsm5.3 T42 fsurdat in namelist_defaults? Remove!
  • Open PR alpha-ctsm5.4.CMIP7.16.ctsm5.3.082: Update namelist_defaults with ctsm54 fsurdat/landuse/finidat files #3576, add new grids to namelist_defaults, rerun ctsm_sci and build-namelist_test.pl
  • Sam R's PR First set of crop parameter changes for CLM6 #3574 after this PR or before? Erik proposed that Sam R. merge to master, and I update the alpha54 branch after
  • New PR: Add ctsm_sci tests for new grids, e.g. NATL, etc. (and new timeperiods, e.g. PtVg, etc.?) instructions here
  • Fix IC files for the new datasets, see updates in ctsm5.4.CMIP7_ciso_ctsm5.3.075_ne30_123_ [AD, SASU, pSASU, HIST] NCAR/LMWG_dev#123 (comment), ctsm5.4.CMIP7_ciso_ctsm5.3.075_f09_124_ [AD, SASU, pSASU, HIST] NCAR/LMWG_dev#124 (comment), 125 is f19 bgc no-crop, 126 is ne30 sp, 127 is f09 sp ...and rerun testing + run all tests
• Verification of the new fsurdat/landuse datasets: Preserve the process in a script. Evaluate all fields in f09 and ne30. Compare ctsm5.4 to 5.3. Try cprnc to compare files and view RMS diffs. See RMS diffs when pft 15/16 LAIs were wrong to help come up with a reasonable alarm. In the unstructured grids we also had the soil texture problem in ctsm5.2/5.3 and fixed it later in ctsm5.3, so use this to come up with an alarm. So far:
  • Step 1: /glade/campaign/cesm/cesmdata/cseg/tools/cime/tools/cprnc/cprnc surfdata_0.9x1.25_hist_1850_78pfts_c251022.nc ../ctsm5.3.0/surfdata_0.9x1.25_hist_1850_78pfts_c240908.nc > surfdata_0.9x1.25_hist_1850_78pfts_c251022.cprnc.out and again for ne30np4.pg3
  • Step 2: ncdiff surfdata_0.9x1.25_hist_1850_78pfts_c251022.nc ../ctsm5.3.0/surfdata_0.9x1.25_hist_1850_78pfts_c240908.nc surfdata_0.9x1.25_hist_1850_78pfts_c251022.dif.nc
  • Step 3: ncview surfdata_0.9x1.25_hist_1850_78pfts_c251022.dif.nc focusing on RMS diffs that seem larger to me; so far all urban (Keith vetted as due to cmip7), const_harvest (Peter: not used), unrep_pft (Peter: cmip7), pct_natveg (Peter: due to the changes in urban), pct_nat_pft (Peter: cmip7), and abm (we updated the code and Fang verified code, fsurdat, and simulation). Some other vars have roundoff diffs. A new ctsm5.3 fsurdat is b4b identical to the official ctsm5.3 fsurdat.
    • Testing with the same number of nodes as used for the ctsm5.4 one, but that's not the culprit (/glade/work/slevis/git_ctsm_tags/ctsm5.3.024/tools/mksurfdata_esmf). So I conclude that the diffs originate in changes in the mksurfdata_esmf code between ctsm5.3 and ctsm5.4 and will not try to identify the exact culprit.
  • Step 4: Compare LAI RMS diffs between surfdata_0.9x1.25_hist_1850_78pfts_c251022.cprnc.out (good) and surfdata_0.9x1.25_hist_2000_78pfts_c250916.cprnc.out (bad pft 15/16). The former is ~roundoff and the latter > 0.1 and normalized > 1. I narrowed down reasonable values with the following grep, where with "E-03" I start to see vars with expected diffs in both files plus LAI in the bad file. Searching for "E+" is already sufficient for picking up the LAI problem:
    grep NORM surfdata_0.9x1.25_hist_1850_78pfts_c251022.cprnc.out | grep 'E-03' same behavior for ne30
  • Step 5: Look at ctsm5.2's soil texture problem: again "E+" reveals unwanted changes in ORGANIC, PCT_SAND, PCT_CLAY, mapunits.
  • Step 6: Starting script that summarizes these steps: Script generates file with grep output. I then use grep to consecutively remove expected contents until I am certain that the file does not contain unexpected fields:
    grep NORM grep_E+_surfdata_cprnc.out | grep -v ROOF | grep -v WALL | grep -v URBAN | grep -v BUILDING | grep -v abm | grep -v CANYON | grep -v CONST_HARVEST | grep -v ROAD | grep -v UNREPRESENTED_PFT | grep -v PCT_NATVEG
  • Add the script to the repository in /tools/mksurfdata_esmf with PR alpha-ctsm5.4.CMIP7.18.ctsm5.3.082: fsurdat validation tool #3612
  • Repeated the mksurfdata_esmf test with 9216 vs. 512 procs here: /glade/derecho/scratch/slevis/temp_work/ctsm5.4/mk_ctsm54_datasets/tools/mksurfdata_esmf
• rimport the new finidat files: /glade/campaign/cesm/cesmdata/cseg/inputdata/list6_ctsm5.4.txt
• rimport the new fsurdat/landuse files: list1 through list5_ctsm5.4.txt
• rimport the raw data, includes
  • new popdens
  • urban properties
  • new abm
  • new landuse
  • new lai, new soil color: see list7_ctsm5.4.txt
  • c13 and c14 handled elsewhere
• Now also need to relink.py these files (list1 through list7)
• Fix FAIL LII2FINIDATAREAS_D_P256x2_Ld1.f09_g17.I1850Clm50BgcCrop.derecho_intel.clm-default COMPARE_base_no_interp #3252 with alpha-ctsm5.4.CMIP7.19.ctsm5.3.082: Fix LII2 COMPARE_base_no_interp failures #3613 and
  • rimport/relink the new finidat
• Finish What's New PR alpha-ctsm5.4.CMIP7.20.ctsm5.3.085: WhatsNewInCTSM5.4.md #3364
• Look over READMEs and other files:
  • READMEs OK
  • PR alpha-ctsm5.4.CMIP7.21.ctsm5.3.085: Three user_nl_clm and two .cfg files with ctsm5.4 updates #3637: Update some test-related files and add ctsm_sci tests to cover the ctsm5.4 simulations listed in What's New
• https://github.com/ESCOMP/CTSM/wiki/Minor-Version-Update-Protocols explains how to compare namelists:
  • This seems ok ./cmp_baseline_lnd_in_files ctsm_sci-ctsm5.3.021 ctsm_sci-alpha-ctsm5.4.CMIP7.19.ctsm5.3.082 > & ! cmp_baseline_lnd_in.021to19.082
  • In the next step, results to investigate:
    • clm6_0 'popdens = clmforc.Li_2025_CMIP7_SSP3CMIP6_hdm_0.5x0.5_simyr1850-2100_c250717.nc'
      This is ok because it replaces the 2018 CMIP6 file for SSP3, which contains identical SSP data.
  • Update wiki instructions to say "rm old lnd_in" files before running comps, and other updates/corrections
• Final PR to merge to main: ctsm5.4.000: Alpha ctsm5.4.cmip7 to master #3638

[samsrabin]

See also: This brief slide deck with some info from Louise Chini; there's also a recording of her talk here.

[wwieder]

And this note from @ckoven on the FATES side

[lawrencepj1]

Hi Everyone
Just for reference this is actually "CMIP6Plus" based on the CMIP Forcings meeting this morning. It is a test version that will be updated if needed to CMIP7 in 2025.

https://wcrp-cmip.org/wp-content/uploads/2024/10/2024-10-29_Forcings-drop-in_session.pdf

That said I am very happy to start going through the LUH3 data and getting CTSM53 raw PFT data generated from the new data.
Thanks
Peter

[olyson]

Since we now have dynamic urban capability, I wonder if we should include urban in this version. I'm not sure what was done with urban previously in CMIP6, maybe it was assigned to some other landunit?

[lawrencepj1]

Hi @olyson

Urban is transient in the LUH data and the values from LUH are explicitly put into the raw CLM5 and CTSM53 PFT data. Mksurfdata however does not use the raw PCT_URBAN data. There is a lot more information required for CTSM than provided by LUH so it would not be useful as a source for final surface or landuse timeseries data.

Cheers
Peter

[olyson]

Thanks @lawrencepj1 . I was thinking that a simple and reasonable thing to do would be to use the PCT_URBAN supplied by LUH and assign it to a single density type (medium density, as that is by far the most common) and then simply splice in the static-in-time urban properties, which don't depend on PCT_URBAN and just depend on density type and region, into those files. We would end up with raw files that have the same information in them as we do now.
If you have a sample raw CLM5/CTSM53 file, I could experiment with that.

[lawrencepj1]

Hi @olyson

Yes not a problem at all. You can find the latest TRENDY and LUH derived CTSM53 raw data at:

/glade/campaign/cesm/development/lmwg/landuse_source_data/CTSM53RawData

SSP5-8.5 has the largest urban expansion of the future scenarios. Here is 2015 vs 2100 values from these two raw files:

globalctsm53TRSSP585Deg025_240728/mksrf_landuse_ctsm53_TRSSP585_2015.c240728.nc
globalctsm53TRSSP585Deg025_240728/mksrf_landuse_ctsm53_TRSSP585_2100.c240728.nc

Peter

[olyson]

Thanks @lawrencepj1 !