FAST.Farm parallel run failure at 11.4 m/s with two turbines – possible OMP solver issue

[linqing-009]

Dear

I am currently performing a dual-turbine floating offshore wind farm simulation using FAST.Farm, and I have encountered an issue that I would like to bring to your attention.

Problem description

Wind farm configuration: two 5 MW floating wind turbines arranged in tandem with a spacing of 6D.
At a wind speed of 8 m/s (uniform or turbulent flow generated by TurbSim), the simulation runs successfully.
When the wind speed is increased to the rated value of 11.4 m/s, the computation produces abnormal results.
Additional finding: With only one turbine in the wind farm, the simulation at 11.4 m/s completes successfully without any anomalies.
Single-turbine OpenFAST simulations under the same conditions run normally, which rules out controller-related issues.
Running FAST.Farm in serial (non-parallel) mode with the same wind farm settings also succeeds, indicating that the .fstf configuration is unlikely to be the cause.
Preliminary assessment​

Based on the above tests, I suspect the problem may be related to the OMP parallel solver, possibly triggering numerical instability or data race conditions under high wind speeds and multi-turbine coupling.

I would greatly appreciate it if you could review the materials and let me know whether there are any known issues related to the parallel solver or suggest possible debugging approaches.

Thank you very much for your time and support.

[linqing-009]

------- FAST.Farm for OpenFAST INPUT FILE -------------------------------------------------
FAST.Farm input file,NREL 5.0 MW Baseline Wind Turbine with OC4-DeepCwind semi configuration, for use in offshore analysis
--- SIMULATION CONTROL ---
False              Echo               - Echo input data to <RootName>.ech? (flag)
FATAL              AbortLevel         - Error level when simulation should abort (string) {"WARNING", "SEVERE", "FATAL"}
3600.0               TMax               - Total run time (s) [>=0.0]
2                  Mod_AmbWind        - Ambient wind model (-) (switch) {1: high-fidelity precursor in VTK format, 2: one InflowWind module, 3: multiple instances of InflowWind module}
2                  Mod_WaveField      - Wave field handling (-) (switch) {1: use individual HydroDyn inputs without adjustment, 2: adjust wave phases based on turbine offsets from farm origin}
0                  Mod_SharedMooring  - Shared mooring system model (switch) {0: None, 3=MoorDyn}}
--- SHARED MOORING SYSTEM --- [used only for Mod_SharedMoor>0]
""                 SharedMoorFile     - Name of file containing shared mooring system input parameters (quoted string) [used only when Mod_SharedMooring > 0]
0.01	           DT_Mooring         - Time step for farm-level mooring coupling with each turbine (s) [used only when Mod_SharedMooring > 0]
true               WrMooringVis       - Write shared mooring visualization, at the global FAST.Farm time step (-) [only used for Mod_SharedMooring=3]
--- AMBIENT WIND: PRECURSOR IN VTK FORMAT --- [used only for Mod_AmbWind=1]
3.0                DT_Low-VTK         - Time step for low -resolution wind data input files; will be used as the global FAST.Farm time step (s) [>0.0]
0.1                DT_High-VTK        - Time step for high-resolution wind data input files (s) [>0.0]
"unused"           WindFilePath       - Path name to VTK wind data files from precursor (string)
False              ChkWndFiles        - Check all the ambient wind files for data consistency? (flag)
--- AMBIENT WIND: INFLOWWIND MODULE --- [used only for Mod_AmbWind=2 or 3]
3.0                DT_Low             - Time step for low -resolution wind data interpolation; will be used as the global FAST.Farm time step (s) [>0.0]
0.5                DT_High            - Time step for high-resolution wind data interpolation (s) [>0.0]
157                NX_Low             - Number  of low -resolution spatial nodes in X direction for wind data interpolation (-) [>=2]
81                 NY_Low             - Number  of low -resolution spatial nodes in Y direction for wind data interpolation (-) [>=2]
31                 NZ_Low             - Number  of low -resolution spatial nodes in Z direction for wind data interpolation (-) [>=2]
-100               X0_Low             - Origin  of low -resolution spatial nodes in X direction for wind data interpolation (m)
-400               Y0_Low             - Origin  of low -resolution spatial nodes in Y direction for wind data interpolation (m)
2.5                Z0_Low             - Origin  of low -resolution spatial nodes in Z direction for wind data interpolation (m)
10                 dX_Low             - Spacing of low -resolution spatial nodes in X direction for wind data interpolation (m) [>0.0]
10.0               dY_Low             - Spacing of low -resolution spatial nodes in Y direction for wind data interpolation (m) [>0.0]
10.0               dZ_Low             - Spacing of low -resolution spatial nodes in Z direction for wind data interpolation (m) [>0.0]
39                 NX_High            - Number  of high-resolution spatial nodes in X direction for wind data interpolation (-) [>=2]
33                 NY_High            - Number  of high-resolution spatial nodes in Y direction for wind data interpolation (-) [>=2]
33                 NZ_High            - Number  of high-resolution spatial nodes in Z direction for wind data interpolation (-) [>=2]
"IW.dat"           InflowFile         - Name of file containing InflowWind module input parameters (quoted string)
--- WIND TURBINES ---
2                  NumTurbines        - Number of wind turbines (-) [>=1]                         [last 6 columns below used only for Mod_AmbWind=2 or 3]
WT_X    WT_Y   WT_Z           WT_FASTInFile            X0_High  Y0_High  Z0_High  dX_High  dY_High  dZ_High
(m)     (m)    (m)              (string)                 (m)      (m)      (m)      (m)      (m)      (m)
 0        0.0     0.0         "5MW_OC4Semi_T1.fst"         -50      -80       5.0        4        5        5
 756      0.0     0.0         "5MW_OC4Semi_T2.fst"         706      -80       5.0        4        5        5
--- WAKE DYNAMICS ---
2                  Mod_Wake           - Switch between wake formulations {1:Polar, 2:Curl, 3:Cartesian} (-) (switch)
126                RotorDiamRef       - Reference turbine rotor diameter for wake calculations (m) [>0.0]
13                 dr                 - Radial increment of radial finite-difference grid (m) [>0.0]
160                NumRadii           - Number of radii in the radial finite-difference grid (-) [>=2]
300                NumPlanes          - Number of wake planes (-) [>=2]
0.2                f_c                - Cutoff (corner) frequency of the low-pass time-filter for the wake advection, deflection, and meandering model [recommended=1.28*U0/R] (Hz) [>0.0] or DEFAULT [DEFAULT=12.5/R, R estimated from dr and NumRadii, not recommended]
DEFAULT            C_HWkDfl_O         - Calibrated parameter in the correction for wake deflection defining the horizontal offset at the rotor                                               (m    ) or DEFAULT [DEFAULT= 0.0  ]
DEFAULT            C_HWkDfl_OY        - Calibrated parameter in the correction for wake deflection defining the horizontal offset at the rotor scaled with                         yaw error (m/deg) or DEFAULT [DEFAULT= 0.0 if Mod_Wake is 2, 0.3 otherwise]
DEFAULT            C_HWkDfl_x         - Calibrated parameter in the correction for wake deflection defining the horizontal offset              scaled with downstream distance               (-    ) or DEFAULT [DEFAULT= 0.0  ]
DEFAULT            C_HWkDfl_xY        - Calibrated parameter in the correction for wake deflection defining the horizontal offset              scaled with downstream distance and yaw error (1/deg) or DEFAULT [DEFAULT= 0.0 if Mod_Wake is 2, -0.004 otherwise]
DEFAULT            C_NearWake         - Calibrated parameter for the near-wake correction (-) [>1.0 and <2.5] or DEFAULT [DEFAULT=1.8]
DEFAULT            k_vAmb        - Calibrated parameters for the influence of the ambient turbulence in the eddy viscosity (set of 5 parameters: k, FMin, DMin, DMax, Exp) (-) [>=0.0, >=0.0 and <=1.0, >=0.0, >DMin, >=0.0] or DEFAULT [DEFAULT=0.05, 1.0, 0.0, 1.0, 0.01]
DEFAULT            k_vShr        - Calibrated parameters for the influence of the shear layer in the eddy viscosity (set of 5 parameters: k, FMin, DMin, DMax, Exp) (-) [>=0.0, >=0.0 and <=1.0, >=0.0, >DMin, >=0.0] or DEFAULT [DEFAULT=0.016, 0.2, 3.0, 25.0, 0.1]
DEFAULT            Mod_WakeDiam       - Wake diameter calculation model (-) (switch) {1: rotor diameter, 2: velocity based, 3: mass-flux based, 4: momentum-flux based} or DEFAULT [DEFAULT=1]
DEFAULT            C_WakeDiam         - Calibrated parameter for wake diameter calculation (-) [>0.0 and <0.99] or DEFAULT [DEFAULT=0.95] [unused for Mod_WakeDiam=1]
DEFAULT            Mod_Meander        - Spatial filter model for wake meandering (-) (switch) {1: uniform, 2: truncated jinc, 3: windowed jinc} or DEFAULT [DEFAULT=3]
2.1                C_Meander          - Calibrated parameter for wake meandering (-) [>=1.0] or DEFAULT [DEFAULT=1.9]
--- CURLED-WAKE PARAMETERS [only used if Mod_Wake=2 or 3] ---
TRUE            Swirl              - Switch to include swirl velocities in wake (-) (switch) [DEFAULT=TRUE]
DEFAULT         k_VortexDecay      - Vortex decay constant for curl (-) [DEFAULT=0.0001]
DEFAULT         NumVortices        - The number of vortices in the curled wake model (-) [DEFAULT=100]
DEFAULT         sigma_D            - The width of the vortices in the curled wake model non-dimesionalized by rotor diameter (-) [DEFAULT=0.2]
DEFAULT         FilterInit         - Switch to filter the initial wake plane deficit and select the number of grid points for the filter {0: no filter, 1: filter of size 1} or DEFAULT [DEFAULT=1] [unused for Mod_Wake=1] (switch)
DEFAULT         k_vCurl            - Calibrated parameter for scaling the eddy viscosity in the curled-wake model (-) [>=0] or DEFAULT [DEFAULT=2.0 ]
DEFAULT         Mod_Projection     - Switch to select how the wake plane velocity is projected in AWAE {1: keep all components, 2: project against plane normal} or DEFAULT [DEFAULT=1: if Mod_Wake is 1 or 3, or DEFAULT=2: if Mod_Wake is 2] (switch)
--- WAKE-ADDED TURBULENCE ---
2                  WAT                - Switch between wake-added turbulence box options {0: no wake added turbulence, 1: predefined turbulence box, 2: user defined turbulence box} (switch)
"../WAT_MannBoxDB/FFDB_D100_512x512x64.u" WAT_BoxFile  - Filepath to the file containing the u-component of the turbulence box (either predefined or user-defined) (quoted string)
512, 512, 64       WAT_NxNyNz         - Number of points in the x, y, and z directions of the WAT_BoxFile [used only if WAT=2, derived value if WAT=1] (-)
5.0, 5.0, 5.0      WAT_DxDyDz         - Distance (in meters) between points in the x, y, and z directions of the WAT_BoxFile [used only if WAT=2, derived value if WAT=1] (m)
default            WAT_ScaleBox       - Flag to scale the input turbulence box to zero mean and unit standard deviation at every node [DEFAULT=False] (flag)
default            WAT_k_Def          - Calibrated parameters for the influence of the maximum wake deficit on wake-added turbulence (set of 5 parameters: k_Def, FMin, DMin, DMax, Exp) (-) [>=0.0, >=0.0 and <=1.0, >=0.0, >DMin, >=0.0] or DEFAULT [DEFAULT=[0.6, 0.0, 0.0, 2.0, 1.0 ]]
default            WAT_k_Grad         - Calibrated parameters for the influence of the radial velocity gradient of the wake deficit on wake-added turbulence (set of 5 parameters: k_Grad, FMin, DMin, DMax, Exp) (-) [>=0.0, >=0.0 and <=1.0, >=0.0, >DMin, >=0.0] or DEFAULT [DEFAULT=[3.0, 0.0, 0.0, 12.0, 0.65]
--- VISUALIZATION ---
False             WrDisWind          - Write low- and high-resolution disturbed wind data to <RootName>.Low.Dis.t<n>.vtk etc.? (flag)
0                  NOutDisWindXY      - Number of XY planes for output of disturbed wind data across the low-resolution domain to <RootName>.Low.DisXY<n_out>.t<n>.vtk (-) [0 to 999]
90.0               OutDisWindZ        - Z coordinates of XY planes for output of disturbed wind data across the low-resolution domain (m) [1 to NOutDisWindXY] [unused for NOutDisWindXY=0]
0                  NOutDisWindYZ      - Number of YZ planes for output of disturbed wind data across the low-resolution domain to <RootName>/Low.DisYZ<n_out>.t<n>.vtk (-) [0 to 999]
126,378,630,882,1134,1386            OutDisWindX        - X coordinates of YZ planes for output of disturbed wind data across the low-resolution domain (m) [1 to NOutDisWindYZ] [unused for NOutDisWindYZ=0]
0                  NOutDisWindXZ      - Number of XZ planes for output of disturbed wind data across the low-resolution domain to <RootName>/Low.DisXZ<n_out>.t<n>.vtk (-) [0 to 999]
30.0                OutDisWindY        - Y coordinates of XZ planes for output of disturbed wind data across the low-resolution domain (m) [1 to NOutDisWindXZ] [unused for NOutDisWindXZ=0]
6.0                WrDisDT            - Time step for disturbed wind visualization output (s) [>0.0] or DEFAULT [DEFAULT=DT_Low or DT_Low-VTK] [unused for WrDisWind=False and NOutDisWindXY=NOutDisWindYZ=NOutDisWindXZ=0]
--- OUTPUT ---
False              SumPrint           - Print summary data to <RootName>.sum? (flag)
99999.9            ChkptTime          - Amount of time between creating checkpoint files for potential restart (s) [>0.0]
0.0                TStart             - Time to begin tabular output (s) [>=0.0]
1                  OutFileFmt         - Format for tabular (time-marching) output file (switch) {1: text file [<RootName>.out], 2: binary file [<RootName>.outb], 3: both}
True               TabDelim           - Use tab delimiters in text tabular output file? (flag) {uses spaces if False}
"ES10.3E2"         OutFmt             - Format used for text tabular output, excluding the time channel.  Resulting field should be 10 characters. (quoted string)
DEFAULT            OutAllPlanes       - Output all wake planes at all time steps. [DEFAULT=False]
0                  NOutRadii          - Number of radial nodes         for wake output for an individual rotor (-) [0 to 20]
0,4,8,12,16,20,24,28,32,36,40,60,79  OutRadii     - List of radial nodes         for wake output for an individual rotor (-) [1 to NOutRadii] [unused for NOutRadii=0]
0                  NOutDist           - Number of downstream distances for wake output for an individual rotor (-) [0 to 9 ]
126,252,378,504,630,756,882  OutDist            - List of downstream distances for wake output for an individual rotor (m) [1 to NOutDist ] [unused for NOutDist =0]
0                  NWindVel           - Number of points for wind output (-) [0 to 9]
0.0                WindVelX           - List of coordinates in the X direction for wind output (m) [1 to NWindVel] [unused for NWindVel=0]
0.0                WindVelY           - List of coordinates in the Y direction for wind output (m) [1 to NWindVel] [unused for NWindVel=0]
90.0               WindVelZ           - List of coordinates in the Z direction for wind output (m) [1 to NWindVel] [unused for NWindVel=0]
                   OutList            - The next line(s) contains a list of output parameters.  See OutListParameters.xlsx for a listing of available output channels (quoted string)
"RtVAmbT1"
"RtVRelT1"
"YawErrT1"
"RtCtAvgT1"
"CtT1N02, CtT1N03, CtT1N04, CtT1N05, CtT1N06"

END of input file (the word "END" must appear in the first 3 columns of this last OutList line)

[andrew-platt]

Which version of FAST.Farm are you using?

We have fixed a few OpenMP related issues in FAST.Farm in recent releases.

[linqing-009]

I’m glad to report that I have successfully used the latest released v5 version, and the turbine can run normally under a uniform rated wind speed of 11.4 m/s. However, when using turbulence wind generated by TurbSim, the calculation still fails: parallel computation produces non-physical, highly oscillatory results—rapid yawing back and forth between 0°, 30°, 90°, 90°, 30°, 0°—and the rotational speed also becomes negative, while serial computation works correctly. The controller has been ruled out as the cause, and I am using DLL files with different naming conventions for the control modules of the two turbines. Could you please advise if there are any other settings I should pay attention to?

[jjonkman]

Dear @linqing-009,

The images you shared in my previous post are not displaying.

Am I correct that you upgraded the input file you previously shared to be compatible with FAST.Farm from OpenFAST v5.0 and you still run into the same issue with the OpenMP version?

Best regards,

[linqing-009]

Dear Mr. Jonkman,

I have been using OpenFAST v5.0, and I am still encountering issues when using OpenMP parallelization. The results are shown in the image below.The same model and turbulent wind field can run successfully in serial computation.

[linqing-009]

Sorry, there was a problem during the image upload process; I have re-uploaded it.

[jjonkman]

Dear @linqing-009,

I'm trying to understand the issue you are running into. If I understand correctly, running with the exactly the same set of input files results in different responses with the serial and OpenMP parallel versions of FAST.Farm within OpenFAST v5; is that correct? Can you share?:

• Your FAST.Farm input file for v5
• Time series results from both turbines for both versions (serial and OpenMP parallel)
• A horizontal slice through the flow field at hub height for both versions

Best regards,

[linqing-009]

Dear @jjonkman ,

You’ve understood perfectly. Below, I provide FAST.Farm input files based on OpenFAST v5. The two sets of input files are completely identical in all configuration parameters, with the only difference being the solution method employed—namely, serial computation versus OpenMP parallel computation.

In addition, I have attached the corresponding time-history results for both methods. Since I am unable to upload the full flow field animation, I randomly captured horizontal cross-sectional images at three different simulation times for comparison. It is noteworthy that, in very rare cases, the exact same input files produce results in parallel computation that are entirely consistent with those from serial computation. However, in most cases, parallel computation exhibits the output characteristics shown below, accompanied by the warning messages illustrated in the figure. What is particularly unusual is that the specific moments at which negative power occurs are not fixed, but appear to be random. At present, the root cause of this phenomenon remains unclear, and I would like to seek your advice and suggestions for further troubleshooting and resolution.

FAST.Farm.txt

[jjonkman]

Dear @linqing-009,

Thanks for clarifying. For the case you shared, it looks like the solutions from the serial and OpenMP versions of FAST.Farm are aligned for the first 300 s or, but then diverge. I'm not sure why that is the case.

I'm curious, if you convert your model to FAST.Farm within OpenFAST v4.2, do you have the same issues?

Best regards,

[andrew-platt]

Dear @linqing-009,

In your FAST.Farm input file for v5.0.0, what values do you have for NumRadii, NumDFull, and NumDBuff? These are on lines 50-52 (https://github.com/OpenFAST/r-test/blob/v5.0.0/glue-codes/fast-farm/TSinflow/FAST.Farm.fstf#L50-L52).
Never mind, I see it in the file you attached above.

Can you include a screenshot of the beginning of the simulation with the compilation info? I'm wondering if this potentially a compiler or platform specific issue, or if it is more general.

Regards,

PS: I edited your second comment with the input file to put the text in a code block quote so it is easier to read.

[linqing-009]

Dear @jjonkman ,

Yes, I initially used the compiler from the v4.2.1 release, and after updating to v5.0, the same type of issue persists.

Dear @andrew-platt ,

I have performed the first few seconds of the simulation and collected the terminal output headers, as shown in the figure. The serial execution (FAST.Farm.exe) shows that all initialization steps are identical to those in the OpenMP parallel run, including:

Sequential execution of ElastoDyn, InflowWind, SeaState, AeroDyn (legacy projection & 2D BEM; unsteady aerodynamics disabled on all nodes/blades due to constant airfoil polars), HydroDyn (reading WAMIT data from marin_semi), and MoorDyn v2.3.8.
MoorDyn configuration: 3 catenary lines, 6 points; all lines initialized as linear due to failed catenary solves, with initial conditions determined through dynamic relaxation. Fairlead tensions converged within 48.6 s.
ServoDyn using the legacy Bladed DLL interface.

In sharp contrast, the OpenMP parallel run begins to exhibit multiple aerodynamic and BEM-related warnings at around 24 s (this moment is not fixed), such as:

BEM solution turned off due to low TSR (approximately 1.75–1.85).
Rotor-averaged axial induction factor > 0.5, leading to limitation of τ1.
No valid flow angle φ for the given local flow conditions.
These issues ultimately result in non-physical behavior: rapid yaw angle oscillations between 0°, 30°, and 90°, and negative power occurring at random times.

The identical initialization confirms that the input files, wind field data, and external controller interfaces are consistent between modes. The emergence of aerodynamic/BEM failures only in the parallel case suggests a parallel-specific issue, likely related to wind field interpolation, time-step synchronization, or inter-thread data handling under transient turbulent inflow.

Best regards.