Tests included in the release test suite

[MakisH]

This issue mainly serves as a discussion for adding more cases in the system tests, starting from the release_test test suite.

The test suite is specified in https://github.com/precice/tutorials/blob/develop/tools/tests/tests.yaml. We currently have:

• elastic-tube-1d
  • {fluid-cpp, solid-python}
• flow-over-heated-plate
  • {fluid-openfoam, solid-nutils}
  • {fluid-openfoam, solid-fenics}
  • {fluid-openfoam, solid-openfoam}

We should still test the following.

First priotiry

These cases are either commonly encountered by users, or they test important features.

• quickstart Add Quickstart to the system tests #771
• perpendicular-flap
  • {fluid-openfoam, solid-openfoam}
  • {fluid-openfoam, solid-calculix}
  • {fluid-openfoam, solid-dealii}
  • {fluid-openfoam, solid-fenics} Add perpendicular-flap solid-openfoam and solid-fenics cases to the system tests #746
  • {fluid-su2, solid-fenics}
• partitioned-heat-conduction
  • {openfoam-dirichlet, openfoam-neumann}
  • {fenics, fenics}
  • fenicsx ?
• flow-over-heated-plate-nearest-projection
  • {fluid-openfoam, solid-openfoam} Add flow-over-heated-plate-nearest-projection case to the system tests #747
• heat-exchanger-simplified
  • {fluid-bottom-openfoam, solid-calculix, fluid-top-openfoam} Add heat-exchanger-simplified to the system tests #731
• multiple-perpendicular-flaps
  • {fluid-openfoam, solid-downstream-dealii, solid-upstream-dealii}
• channel-transport
  • {fluid-nutils, transport-nutils}
• flow-over-heated-plate-two-meshes
  • {fluid-openfoam, solid-calculix}
• partitioned-heat-conduction-complex
  • {fenics, fenics}
• partitioned-heat-conduction-direct
  • {nutils, nutils}
• volume-coupled-diffusion
  • {fenics, fenics}
• aste-turbine

Later

• perpendicular-flap
  • {fluid-openfoam, solid-dune}
  • {fluid-openfoam, solid-solids4foam}
  • {fluid-su2, solid-calculix}
• flow-over-heated-plate
  • {fluid-openfoam, solid-dunefem}
• channel-transport-reaction
  • {fluid-fenics, chemical-fenics}
• flow-over-heated-plate-steady-state
  • {fluid-openfoam, solid-codeaster}
• oscillator
  • {python, python}
• partitioned-elastic-beam
  • {dirichlet-calculix, neumann-calculix}

Very long cases

We don't want to test very long cases yet. The plan is to eventually restrict them to some first time steps (#402). These are:

• elastic-tube-3d
• heat-exchanger
• turek-hron-fsi3
• Anything with a fluid Nutils solver (e.g., perpendicular-flap {fluid-nutils, solid-calculix})

Not relevant anymore

Cannot be tested in v2

We cannot compare reference results for cases that were not there in the previous distribution release or that had different setups:

• channel-transport
  • {fluid-openfoam, transport-nutils}
• flow-over-heated-plate-partitioned-flow
  • {fluid1-openfoam, fluid2-openfoam, solid-openfoam}
• partitioned-backwards-facing-step
  • {fluid1-openfoam, fluid2-openfoam}
• partitioned-pipe-two-phase
  • {fluid1-openfoam, fluid2-openfoam}
• partitioned-pipe (boundary conditions changed)
  • {fluid1-openfoam-pimplefoam, fluid2-openfoam-pimplefoam}
  • {fluid1-openfoam-sonicliquidfoam, fluid2-openfoam-sonicliquidfoam}
• two-scale-heat-conduction
  • {macro-nutils, micro-nutils}
• volume-coupled-flow
  • {source-nutils, fluid-openfoam}

[MakisH]

We should also make a table of test cases x components/features, to discover blind spots on what we are not testing.

[MakisH]

First priority (currently missing) should be the Quickstart, since this is the first case users run. We can handle it similarly to the nutils tutorials.

[MakisH]

@fsimonis also wanted to have the solverdummies included here.

[uekerman]

Documentation on how to add testcases and components: https://precice.org/dev-docs-system-tests.html

[nithin434]

Hi! I’m exploring this issue for a potential GSoC proposal and would like to start with a well-scoped, high-impact contribution. My initial goal is to add the Quickstart tutorial as a system test following the developer documentation, including generating reference results and automated comparison. After that, I’m interested in extending the work toward a feature–test coverage table to understand what is currently not tested.
Please let me know if this is a good starting point or if there’s a specific missing case you’d prefer I focus on first.
@MakisH @fsimonis

[MakisH]

Welcome, @nithin434! Adding the Quickstart would be great. That tutorial would also require thinking what to do with the solid-cpp, which does not use an adapter. You could treat it similarly to the elastic-tube-1d.

Since most of the work here would be running the tests and seeing that the these work, I would appreciate some logs in the respective PR. The CI does not currently let us test changes to the system tests themselves submitted from forks.

[nithin434]

Thanks for the pointers! I’ll start working on adding the Quickstart as a system test and treat the solid-cpp case similar to elastic-tube-1d. I’ll run everything locally and include the logs in the PR. I’ll open a draft PR soon for early feedback @MakisH