conda fails to load for SystemTests

[samsrabin]

Brief summary of bug

For some users, FSURDATMODIFYCTSM is failing because conda (called in a subprocess from Python) doesn't load. Note that this is a different issue (and gives a different sort of error) from #2109.

General bug information

CTSM version you are using: ctsm5.1.dev133-47-g6925f8cc9

Does this bug cause significantly incorrect results in the model's science? No

Configurations affected: FSURDATMODIFYCTSM and RXCROPMATURITY tests.

Details of bug

@rgknox and @ekluzek (#1959), as well as @slevis-lmwg (#2106), are all affected. This doesn't happen for me or, I think, @adrifoster. I think it's thus something to do with our shell environments.

We suspect this stems from changes introduced in cime_config/SystemTests/ in ctsm5.1.dev131 (as was the case for #2109). The module unload python; module load conda; step fails to load conda, so then the call of conda run -n fails and ends the test. However, that module un/loading step was present before, so there's something subtle going on.

Important details of your setup / configuration so we can reproduce the bug

Unknown.

Important output or errors that show the problem

From TestStatus.log:

which: no conda in (/glade/u/apps/ch/opt/netcdf/4.9.0/intel/19.1.1/bin:/glade/u/apps/ch/opt/ncarcompilers/0.5.0/intel/19.1.1:/glade/u/apps/ch/opt/ncarcompilers/0.5.0/intel/19.1.1/mpi:/glade/p/cesmdata/cseg/PROGS/esmf/8.4.1b02/mpiuni/2.3.0/intel/19.1.1/bin/bing/Linux.intel.64.mpiuni.default:/glade/u/apps/opt/intel/2020u1/vtune_profiler/bin64:/glade/u/apps/opt/intel/2020u1/inspector/bin64:/glade/u/apps/opt/intel/2020u1/advisor/bin64:/glade/u/apps/opt/intel/2020u1/compilers_and_libraries/linux/bin/intel64:/glade/u/apps/ch/opt/cmake/3.22.0/bin:/glade/u/apps/ch/opt/python/3.7.9/gnu/9.1.0/bin:/glade/u/apps/opt/vncmgr:/glade/u/apps/opt/globus-utils:/glade/u/apps/ch/opt/usr/bin:/glade/u/apps/ch/opt/lmod/8.7.13/bin:/glade/u/apps/opt/intel/2020u1/vtune_amplifier/bin64:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/bin/X11:/usr/games:/opt/ibutils/bin:/usr/lib/mit/bin:/opt/pbs/bin:/glade/u/apps/opt/hpss/hpss-client.5.0.2.p5/bin:/glade/u/apps/ch/os/usr/bin:/glade/u/apps/ch/opt/hpe-cpe/22.02/bin)
Traceback (most recent call last):
  File "./case.build", line 256, in <module>
    _main_func(__doc__)
  File "./case.build", line 216, in _main_func
    test = find_system_test(testname, case)(case)
  File "/glade/u/home/rgknox/ctsm/cime_config/SystemTests/fsurdatmodifyctsm.py", line 46, in __init__
    self._run_modify_fsurdat()
  File "/glade/u/home/rgknox/ctsm/cime_config/SystemTests/fsurdatmodifyctsm.py", line 77, in _run_modify_fsurdat
    tool_path,
  File "/glade/u/home/rgknox/ctsm/cime_config/SystemTests/systemtest_utils.py", line 39, in run_python_script
    command, shell=True, check=True, text=True, stdout=f, stderr=subprocess.STDOUT
  File "/glade/u/apps/ch/opt/python/3.7.9/gnu/9.1.0/lib/python3.7/subprocess.py", line 512, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '. /glade/scratch/rgknox/tests_0814-134713ch/FSURDATMODIFYCTSM_D_Mmpi-serial_Ld1.5x5_amazon.I2000Clm50SpRs.cheyenne_intel.GC.0814-134713ch_int/.env_mach_specific.sh;  module unload python; module load conda; conda run -n ctsm_pylib python3 /glade/u/home/rgknox/ctsm/tools/modify_input_files/fsurdat_modifier /glade/scratch/rgknox/tests_0814-134713ch/FSURDATMODIFYCTSM_D_Mmpi-serial_Ld1.5x5_amazon.I2000Clm50SpRs.cheyenne_intel.GC.0814-134713ch_int/modify_fsurdat.cfg' returned non-zero exit status 127.

And fsurdat_modifier.log:

Lmod has detected the following error: Unable to load module because of error
when evaluating modulefile:
     /glade/u/apps/ch/modulefiles/default/idep/conda/latest.lua: [string
"require("posix")..."]:54: attempt to call global 'source_sh' (a nil value)
     Please check the modulefile and especially if there is a the line number
specified in the above message
While processing the following module(s):
    Module fullname  Module Filename
    ---------------  ---------------
    conda/latest     /glade/u/apps/ch/modulefiles/default/idep/conda/latest.lua

/bin/sh: conda: command not found

[ekluzek]

Note this is something that @rgknox and I see, but @samsrabin and @adrifoster don't? So works for some and not for others. Still a good thing to figure out.

Also there error checking for these situations could be better so that it's more obvious what is going on.

[billsacks]

I am seeing this error in my testing for ctsm5.1.dev136. The test FSURDATMODIFYCTSM_D_Mmpi-serial_Ld1.5x5_amazon.I2000Clm50SpRs.cheyenne_intel fails in the SHAREDLIB_BUILD phase.

[adrifoster]

This was the same error I got because I had a conda environment loaded Can you try doing "conda deactivate" and then rebuilding?

…

On Tue, Aug 22, 2023 at 11:34 AM Bill Sacks ***@***.***> wrote: I am seeing this error in my testing for ctsm5.1.dev136. The test FSURDATMODIFYCTSM_D_Mmpi-serial_Ld1.5x5_amazon.I2000Clm50SpRs.cheyenne_intel fails in the SHAREDLIB_BUILD phase. — Reply to this email directly, view it on GitHub <#2111 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADE42IWRH7XJG3QUJOIZW4TXWTUUBANCNFSM6AAAAAA3SUUV7Y> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[samsrabin]

@adrifoster I think your error was actually the one given in #2109, no?

[samsrabin]

@ekluzek and I worked on this a few days ago, and the conclusion (among other notes) was that there's probably something up with his environment. This may be biting all users with old Cheyenne accounts, as Adrianna and I don't experience the issue.

This may end up taking a long time to debug. In the interim, I think I'll add a fallback to the original behavior if the new behavior fails.

[billsacks]

This was the same error I got because I had a conda environment loaded

I don't think this was the case for me.

I agree with the idea of doing a quick-ish workaround for now, not spending a lot of time on this. Time would probably better be spent figuring out what needs to be done to get this working on derecho, if it doesn't work out-of-the-box.

[samsrabin]

A perplexing new development from some troubleshooting just now with @slevis-lmwg… I had him use the cime_config/SystemTests/fsurdatmodifyctsm.py from tag ctsm5.1.dev129—i.e., bypassing all my changes to how the test is run—and it still failed with the same "conda not found" error! It seems like something else might be causing the issue.

@billsacks @ekluzek @rgknox: Would any of y'all be willing to try this to confirm?