Archive fieldcompare diffs only on failure, use diff-results/ and extension-specific globs

Author: PranjalManhgaye

changelog-entries/441.md

@@ -1 +1 @@
-- Archive fieldcompare diff VTK files into a `fieldcompare-diffs/` folder in each systemtest run directory so they are easy to find in CI artifacts when investigating comparison failures (fixes [#441](https://github.com/precice/tutorials/issues/441)).
+- Archive fieldcompare diff VTK files into a `diff-results/` folder in each systemtest run directory on failure so they are easy to find in CI artifacts when investigating comparison failures (fixes [#441](https://github.com/precice/tutorials/issues/441)).

tools/tests/README.md

@@ -105,7 +105,7 @@ In this case, building and running seems to work out, but the tests fail because
 
 The easiest way to debug a systemtest run is first to have a look at the output written into the action on GitHub.
 If this does not provide enough hints, the next step is to download the generated `system_tests_run_<run_id>_<run_attempt>` artifact. Note that by default this will only be generated if the systemtests fail.
-Inside the archive, a test-specific subfolder like `flow-over-heated-plate_fluid-openfoam-solid-fenics_2023-11-19-211723` contains two log files: a `stderr.log` and `stdout.log`. This can be a starting point for a further investigation. When fieldcompare runs with `--diff`, it writes VTK diff files; these are copied into a `fieldcompare-diffs/` subfolder in the same run directory so you can open them (e.g. in ParaView) to see where results differ from the reference.
+Inside the archive, a test-specific subfolder like `flow-over-heated-plate_fluid-openfoam-solid-fenics_2023-11-19-211723` contains two log files: a `stderr.log` and `stdout.log`. This can be a starting point for a further investigation. When fieldcompare runs with `--diff`, it writes VTK diff files; these are copied into a `diff-results/` subfolder in the same run directory so you can open them (e.g. in ParaView) to see where results differ from the reference.
 
 ## Adding new tests
 

tools/tests/systemtests/Systemtest.py

@@ -6,7 +6,7 @@
 from pathlib import Path
 from paths import PRECICE_REL_OUTPUT_DIR, PRECICE_TOOLS_DIR, PRECICE_REL_REFERENCE_DIR, PRECICE_TESTS_DIR, PRECICE_TUTORIAL_DIR
 
-FIELDCOMPARE_DIFFS_DIR = "fieldcompare-diffs"
+DIFF_RESULTS_DIR = "diff-results"
 
 from metadata_parser.metdata import Tutorial, CaseCombination, Case, ReferenceResult
 from .SystemtestArguments import SystemtestArguments
@@ -417,16 +417,18 @@ def _run_field_compare(self):
 
     def __archive_fieldcompare_diffs(self):
         """
-        Copy fieldcompare diff VTK files from precice-exports into fieldcompare-diffs/
-        so they are easy to find in CI artifacts (issue #441).
+        Copy fieldcompare diff VTK files from precice-exports into diff-results/
+        so they are easy to find in CI artifacts when investigating failures (issue #441).
         """
         precice_exports = self.system_test_dir / PRECICE_REL_OUTPUT_DIR
         if not precice_exports.exists():
             return
-        diff_files = list(precice_exports.glob("diff_*")) + list(precice_exports.glob("*_diff.*"))
+        diff_files = []
+        for pattern in ("*diff*.vtu", "*diff*.vtk", "*diff*.vtp"):
+            diff_files.extend(precice_exports.glob(pattern))
         if not diff_files:
             return
-        dest_dir = self.system_test_dir / FIELDCOMPARE_DIFFS_DIR
+        dest_dir = self.system_test_dir / DIFF_RESULTS_DIR
         dest_dir.mkdir(exist_ok=True)
         for f in diff_files:
             if f.is_file():
@@ -583,10 +585,10 @@ def run(self, run_directory: Path):
                 fieldcompare_time=0)
 
         fieldcompare_result = self._run_field_compare()
-        self.__archive_fieldcompare_diffs()
         std_out.extend(fieldcompare_result.stdout_data)
         std_err.extend(fieldcompare_result.stderr_data)
         if fieldcompare_result.exit_code != 0:
+            self.__archive_fieldcompare_diffs()
             self.__write_logs(std_out, std_err)
             logging.critical(f"Fieldcompare returned non zero exit code, therefore {self} failed")
             return SystemtestResult(