Ensemble filename and stoch restart fixes by alperaltuntas · Pull Request #409 · NCAR/MOM6

alperaltuntas · 2026-02-27T02:58:16Z

This PR brings in the following changes:

Fix stoch restart file handling:
- append ensemble instance suffix in multi-instance runs
- add r_stoch file(s) to CESM rpointer files.
Introduce append_ensemble_appendix: To eliminate code duplication, this commit introduces append_ensemble_appendix, a routine to append the ensemble appendix to i/o filenames. If provided,
the appendix is appended after the last occurence of ENSEMBLE_APPENDIX_PREFIX,
a new runtime parameter (initially called RESTARTFILE_APPENDIX_PREFIX).

This PR is part of a series of PRs to fix various interrelated restart/multi-instance/hybrid run issues alongside: ESCOMP/FMS#8
ESCOMP/MOM_interface#311
ESMCI/cime#4944

To eliminate code duplication, this commit introduces append_ensemble_appendix, a routine to append the ensemble appendix to i/o filenames. If provided, the appendix is appended after the last occurence of ENSEMBLE_APPENDIX_PREFIX, a new runtime parameter (initially called RESTARTFILE_APPENDIX_PREFIX).

- append ensemble instance suffix in multi-instance runs - add it to CESM rpointer files.

iangrooms · 2026-03-01T15:39:24Z

Thanks @alperaltuntas for working on this. @kateboden ran a quick test using this PR code; results can be found at /glade/derecho/scratch/kboden/g.e30_a07g.G_JRA.TL319_t232_zstar_N75.2025.SKEB_DA.restart_test/.
Ensemble member 0040 (for example) writes two restarts with names

g.e30_a07g.G_JRA.TL319_t232_zstar_N75.2025.SKEB_DA.restart_test.mom6.r.0054-01-06-00000_0040.nc
g.e30_a07g.G_JRA.TL319_t232_zstar_N75.2025.SKEB_DA.restart_test.mom6_0040.r_stoch.0054-01-06-00000.nc

This is an improvement because previously the stochastic-physics restart file didn't have an instance number at all, but the location of the instance number in the stochastic-physics restart filename is inconsistent with the mom6 restart filename.

The rpointer file is named rpointer.ocn_0040.0054-01-06-00000 and its contents are

g.e30_a07g.G_JRA.TL319_t232_zstar_N75.2025.SKEB_DA.restart_test.mom6_0040.r.0054-01-06-00000.nc
g.e30_a07g.G_JRA.TL319_t232_zstar_N75.2025.SKEB_DA.restart_test.mom6_0040.r_stoch.0054-01-06-00000.nc

So the rpointer file contents are not consistent with the actual restart names. We have not tried to restart from these files yet.

This may be out of scope, but the instance number is placed inconsistently in some other filenames too. For example:

g.e30_a07g.G_JRA.TL319_t232_zstar_N75.2025.SKEB_DA.restart_test.mom6.h.sfc._0040.0054-01.nc
g.e30_a07g.G_JRA.TL319_t232_zstar_N75.2025.SKEB_DA.restart_test.mom6.h.static._0040.nc
g.e30_a07g.G_JRA.TL319_t232_zstar_N75.2025.SKEB_DA.restart_test.mom6.h.z._0040.0054-01.nc
g.e30_a07g.G_JRA.TL319_t232_zstar_N75.2025.SKEB_DA.restart_test.mom6.ic.0054-01-01_0040.nc
g.e30_a07g.G_JRA.TL319_t232_zstar_N75.2025.SKEB_DA.restart_test.mom6.h.Fram_Strait._0040.0054-01.nc.0856
ocean.stats._0040.nc
ocean.stats._0040

The history files all have it before the date while the ic file has it after the date. This is just aesthetic, so not a high priority.

alperaltuntas · 2026-03-01T18:02:34Z

Thanks @iangrooms. If you haven't already, could you re-run these tests after merging all of the other PRs too (mentioned in the PR description):

ESCOMP/FMS#8
ESCOMP/MOM_interface#311
ESMCI/cime#4944

For convenience, I have a sandbox on derecho, where all these PRs are applied: /glade/work/altuntas/cesm.sandboxes/cesm3_0_alpha08d_feb27

iangrooms · 2026-03-02T16:56:41Z

@kateboden ran a second test using your sandbox; results can be found at /glade/derecho/scratch/kboden/g.e30_a07g.G_JRA.TL319_t232_zstar_N75.2025.SKEB_DA.restart_test_2/. The instance number is in the right place in all the filenames.

mnlevy1981

A few comments on the names of variables / functions; if you were just following how similar objects are named elsewhere in the code, I'm happy to approve as-is. (I didn't do any testing, but saw other comments about file names being correct in other peoples' tests)

mnlevy1981 · 2026-03-03T19:50:41Z

+!> Append the ensemble appendix to a filename. If provided, the appendix is appended after
+!! the last occurrence of the append_after substring in the filename.
+subroutine append_ensemble_appendix(filename, append_after)


Could we rename this function insert_ensemble_id() or something in that vein? I mostly take issue with the append_ portion, because we are not always appending this string; I see that MOM and FMS use appendix and suffix to refer to the ensemble member identifier.

Sounds good!

I ended up renaming it as insert_ensemble_appendix, assuming that id refers to i while appendix refers to _000i for the i'th component (where 0<=i<=9)

mnlevy1981 · 2026-03-03T19:52:29Z

+  filename_appendix_t = trim(filename_appendix_t)
+  if (len(filename_appendix_t) == 0) return


Instead of trimming the string and then finding the length, you could just check (len_trim(filename_appendix_t) == 0)

Here, I actually switched from filename_appendix_t -> filename_appendix and used len_trim since filename_appendix_t wasn't a deferred-length string, unlike the other two.

mnlevy1981 · 2026-03-03T19:53:21Z

+  filename_t = trim(filename)
+  pos = len(filename_t)


Similarly, you can just use pos = len_trim(filename_t); you are going to change filename_t below anyway, so the assignment line isn't necessary

I added _t versions of these strings to prevent multiple calls and accidental omissions of trim() in the rest of the function. In fact, I think I should have:

filename_t = trim(adjustl(filename))

If you think _t versions of these strings cause unnecessary confusion, I am fine with removing them.

(Just noticed that trim call in 3039 is unnecessary, so it needs to be removed at the least.)

I managed to miss that 3018 was filename_t on the lefthand side and filename on the right; I think your comment about using adjustl() makes more sense than dropping filename_t -- I noticed the trim(adjustl()) construct elsewhere in MOM6 when looking for where GRID_FILE is read in.

mnlevy1981 · 2026-03-03T19:54:31Z

+      call MOM_error(FATAL, "append_ensemble_appendix: The string "//trim(append_after)// &
+                     " was not found in the filename "//trim(filename))
+    endif
+    pos = pos + len_trim(append_after_t) - 1


Opposite comment time :) Since append_after_t is deferred-length and is trim(append_after), don't you want len(append_after_t) instead of len_trim()?

Right, this should be just len(append_after_t). I think I initially had:

len_trim(append_after)

then changed append_after to append_after_t, but didn't update len_trim to len.

mnlevy1981 · 2026-03-03T19:59:49Z

 public ocean_public_type_chksum
 public get_ocean_grid, query_ocean_state
 public get_eps_omesh
+public stoch_restart_needed


I really like having this as a stand-alone function!

iangrooms

I don't have any comments about the implementation, but in our ensemble data assimilation test this works.

alperaltuntas · 2026-03-05T18:26:53Z

@mnlevy1981 I believe I addressed all your comments, but let me know otherwise. Thanks!

mnlevy1981

Thanks for making those changes, this looks great!

alperaltuntas added 2 commits February 26, 2026 19:45

Fix stoch restart file handling:

c4ef953

- append ensemble instance suffix in multi-instance runs - add it to CESM rpointer files.

alperaltuntas requested a review from mnlevy1981 February 27, 2026 03:01

alperaltuntas force-pushed the restart_file_fixes branch from 08a38c5 to c4ef953 Compare February 27, 2026 03:10

This was referenced Feb 27, 2026

Updates and fixes for ensemble runs ESCOMP/MOM_interface#311

Merged

Skip MOM6 .ic. history file comparison in multi-instance runs ESMCI/cime#4944

Merged

Append ensemble suffixes right after .mom6 substring in history files ESCOMP/FMS#8

Merged

alperaltuntas requested a review from iangrooms February 27, 2026 17:19

mnlevy1981 requested changes Mar 3, 2026

View reviewed changes

iangrooms approved these changes Mar 5, 2026

View reviewed changes

address reviews

26648d4

mnlevy1981 approved these changes Mar 10, 2026

View reviewed changes

alperaltuntas merged commit b15bc97 into NCAR:dev/ncar Mar 12, 2026
52 checks passed

chengz2 mentioned this pull request Apr 22, 2026

NCAR to main (2026-04-22) mom-ocean/MOM6#1700

Draft

		filename_appendix_t = trim(filename_appendix_t)
		if (len(filename_appendix_t) == 0) return

Conversation

alperaltuntas commented Feb 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

iangrooms commented Mar 1, 2026

Uh oh!

alperaltuntas commented Mar 1, 2026

Uh oh!

iangrooms commented Mar 2, 2026

Uh oh!

mnlevy1981 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alperaltuntas Mar 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

iangrooms left a comment

Choose a reason for hiding this comment

Uh oh!

alperaltuntas commented Mar 5, 2026

Uh oh!

mnlevy1981 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

alperaltuntas commented Feb 27, 2026 •

edited

Loading

alperaltuntas Mar 3, 2026 •

edited

Loading