Regression tests output doesn't show which testcase is being run and for which scenario (1D/2D)

[niketagrawal]

Current behavior

The regression tests implemented in aeolis/tests/regression_tests/ checks whether whether running the simulation for the following cases produces the same netCDF file consistently across other code changes in the repository

• 1D/case1_small_waves
• 1D/case2_larger_waves
• 1D/case3_erosion_avalanching
• 2D/Barchan_dune

The testcases include

• check whether netCDF file is created as part of the simulation
• check whether aeolis.log file created as part of the simulation
• check whether the array shape, dimension, and array values in the netCDF file produced are the same as the ones stored in a reference output for the same model configuration file.

Currently, the pytest output doesn't show the pass/fail status of testcase for each of the cases making it difficult to understand the test output and debug it in case of a failure.

Desired behavior

Display pass/fail status of each testcase per scenario in the pytest output.

Fix

• The desired behavior can be achieved by breaking the large test into individual testcases and using parametrization.
• A proof of concept is implemented in the branch 226-parametrize-regression-tests
• Check out the above branch and execute pytest -v on the command line.
• The pytest output that this implementation produces is the following:

aeolis/tests/regression_tests/test_simulation.py::TestOutputFileGeneration::test_netCDF_creation[1D_case1_small_waves] PASSED                                         [  5%]
aeolis/tests/regression_tests/test_simulation.py::TestOutputFileGeneration::test_log_file_creation[1D_case1_small_waves] PASSED                                       [ 10%]
aeolis/tests/regression_tests/test_simulation.py::TestnetCDFContent::test_array_shape[1D_case1_small_waves] PASSED                                                    [ 15%]
aeolis/tests/regression_tests/test_simulation.py::TestnetCDFContent::test_array_dimension[1D_case1_small_waves] PASSED                                                [ 20%]
aeolis/tests/regression_tests/test_simulation.py::TestnetCDFContent::test_array_values[1D_case1_small_waves] FAILED                                                   [ 25%]
aeolis/tests/regression_tests/test_simulation.py::TestOutputFileGeneration::test_netCDF_creation[1D_case2_larger_waves] PASSED                                        [ 30%]
aeolis/tests/regression_tests/test_simulation.py::TestOutputFileGeneration::test_log_file_creation[1D_case2_larger_waves] PASSED                                      [ 35%]
aeolis/tests/regression_tests/test_simulation.py::TestnetCDFContent::test_array_shape[1D_case2_larger_waves] PASSED                                                   [ 40%]
aeolis/tests/regression_tests/test_simulation.py::TestnetCDFContent::test_array_dimension[1D_case2_larger_waves] PASSED                                               [ 45%]
aeolis/tests/regression_tests/test_simulation.py::TestnetCDFContent::test_array_values[1D_case2_larger_waves] FAILED                                                  [ 50%]
aeolis/tests/regression_tests/test_simulation.py::TestOutputFileGeneration::test_netCDF_creation[1D_case3_erosion_avalanching] ERROR                                  [ 55%]
aeolis/tests/regression_tests/test_simulation.py::TestOutputFileGeneration::test_log_file_creation[1D_case3_erosion_avalanching] ERROR                                [ 60%]
aeolis/tests/regression_tests/test_simulation.py::TestnetCDFContent::test_array_shape[1D_case3_erosion_avalanching] ERROR                                             [ 65%]
aeolis/tests/regression_tests/test_simulation.py::TestnetCDFContent::test_array_dimension[1D_case3_erosion_avalanching] ERROR                                         [ 70%]
aeolis/tests/regression_tests/test_simulation.py::TestnetCDFContent::test_array_values[1D_case3_erosion_avalanching] ERROR


========================================================================== short test summary info ==========================================================================
FAILED aeolis/tests/regression_tests/test_simulation.py::TestnetCDFContent::test_array_values[1D_case1_small_waves] - AssertionError: Array values of the parameter 'zs' are expected to remain consistent across simulations for the same model parameter file for 1D case1_small_waves
FAILED aeolis/tests/regression_tests/test_simulation.py::TestnetCDFContent::test_array_values[1D_case2_larger_waves] - AssertionError: Array values of the parameter 'zs' are expected to remain consistent across simulations for the same model parameter file for 1D case2_larger_waves
FAILED aeolis/tests/regression_tests/test_simulation.py::TestnetCDFContent::test_array_values[2D_Barchan_dune] - AssertionError: Array values of the parameter 'time' are expected to remain consistent across simulations for the same model parameter file for 2D Barchan_dune
ERROR aeolis/tests/regression_tests/test_simulation.py::TestOutputFileGeneration::test_netCDF_creation[1D_case3_erosion_avalanching] - KeyError: 'Cs'
ERROR aeolis/tests/regression_tests/test_simulation.py::TestOutputFileGeneration::test_log_file_creation[1D_case3_erosion_avalanching] - KeyError: 'Cs'
ERROR aeolis/tests/regression_tests/test_simulation.py::TestnetCDFContent::test_array_shape[1D_case3_erosion_avalanching] - KeyError: 'Cs'
ERROR aeolis/tests/regression_tests/test_simulation.py::TestnetCDFContent::test_array_dimension[1D_case3_erosion_avalanching] - KeyError: 'Cs'
ERROR aeolis/tests/regression_tests/test_simulation.py::TestnetCDFContent::test_array_values[1D_case3_erosion_avalanching] - KeyError: 'Cs'
===================================================== 3 failed, 34 passed, 5004 warnings, 5 errors in 125.60s (0:02:05) =====================================================

[Sierd]

@niketagrawal, this works well. I committed some changes to the branch to make the tests pass (mainly updating the expected outputs. I don't think we should test for case 3. The erosion module seems a bit buggy still. @ncohn maybe you can comment, is the erosion module robust enough for running generic tests?

[niketagrawal]

@Sierd thanks for your inputs.

Two test cases fail due to changes introduced in the latest commit 39681da

aeolis/tests/regression_tests/test_simulation.py::TestnetCDFContent::test_array_values[1D_case1_small_waves] FAILED  
aeolis/tests/regression_tests/test_simulation.py::TestnetCDFContent::test_array_values[1D_case2_larger_waves] FAILED         

Reason for test cases failure
Array values for the parameter y in the netcdf generated is different than the ones in the reference output file. See the comparison below

Parameter: y
Actual values for 1D case1_small_waves:
[[0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5
  0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5
  0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5
  0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5
  0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5
  0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5
  0.5 0.5 0.5 0.5 0.5 0.5]]
Expected values for 1D case1_small_waves:
[[0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
  0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
  0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
  0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
  0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]]

Parameter: y
Actual values for 1D case2_larger_waves:
[[0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5
  0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5
  0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5
  0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5
  0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5
  0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5
  0.5 0.5 0.5 0.5 0.5 0.5]]
Expected values for 1D case2_larger_waves:
[[0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
  0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
  0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
  0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
  0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]]

Does the latest commit 39681da alters how the array values of the parameter y are calculated than how it was done previously? If yes, then replacing the reference output file will fix this issue.