`FAIL_FAST_ALWAYS` for selection of `DockerClientProviderStrategy` not reset in flaky environments when tests are retried

[arhont375]

Hello team!
Please provide ability to disable FAIL_FAST_ALWAYS usage for the selection of DockerClientProviderStrategy located here:

testcontainers-java/core/src/main/java/org/testcontainers/dockerclient/DockerClientProviderStrategy.java

Line 55 in abf1846

private static final AtomicBoolean FAIL_FAST_ALWAYS = new AtomicBoolean(false);

Our use case:

• We have big mono repo which periodically execute "test all" kind of job
• This job running gradle tests in parallel (via gradle settings) and a lot of projects using test containers to perform various tests
• Possibly due to the high usage of docker (this part hard to investigate well for me) sometimes test containers will not be able to find docker environment even though it's present and will set flag FAIL_FAST_ALWAYS
• As we use following plugin https://github.com/gradle/test-retry-gradle-plugin that retry tests as part of the same task I assume that initialized static field is not reset when it retry failed tests. So this plugin trying to retry test from the same task for test that uses test containers and it will fail due to FAIL_FAST_ALWAYS flag be set

Having ability to disable this flag on demand or being able to forcibly configure retries even if that flag is set would be helpful in our situation

[kiview]

Hi @arhont375, thanks for reporting this issue and sharing the details of your Testcontainers setup. I am not yet sure if we should follow up on your suggested implementation, or need to look at the root cause first to make the code generally more robust, but I can understand that a case of resource contention together with the Test Retry Gradle plugin can lead to this failure scenario.

[arhont375]

Thank you for quick response!
Please let me know if there is something that I can help with

[kiview]

Thanks for the offer @arhont375. Do you have a way to reliably reproduce your issues? In this case, I might come back to you for trying out a PR to mitigate this situation, once we come around to tackling this.

[kiview]

One question I forgot to ask you @arhont375, can you describe your setup? Is this happening locally or on CI? How is Docker installed?

[arhont375]

One question I forgot to ask you @arhont375, can you describe your setup? Is this happening locally or on CI? How is Docker installed?

This reproducible locally (ubuntu, system installation of docker) and in the CI (Jenkins + CentoOS + system installation of docker).

This may depend on machine that execute it, but following repo allow to reproduce it on my small machine (2 cores, 4 threads, Intel Core i7): https://github.com/arhont375/test-containters-5017
Even if it will not fail for you it may give picture about how tests executed in our environment: parallel execution of projects in gradle + parallel execution of tests via JUnit + many short living tests using containers.
PS
I added gradle-test-retry plugin there, but somehow not sure if it set up properly.

[kiview]

Thanks, this information helps a lot, because this shows it happens on a native Linux Docker installation, rather than Docker Desktop (on Mac or Windows), which would further complicate debugging.

So we might see an overloaded single Docker daemon in your case (which might lead to errors when being pinged).

[hadson172]

Having same issue. Any help on this?

[arhont375]

Having same issue. Any help on this?

Don't work there anymore, but I think we forcibly redefined flag via reflection.

[hinst]

I have encountered this problem recently.
When running unit tests in parallel mode, I sometimes get "Could not find a valid Docker environment" error.
Using Windows 11 and Docker Desktop.
Unfortunately I have no idea what causes this error, because no further information is printed into log in my case.
The ideal solution would be to find the root cause, however it seems not possible because of missing explanation, so retry is the only solution for now. Please make it possible to disable FAIL_FAST_ALWAYS
I have already confirmed that retry is helpful in my case, by resetting FAIL_FAST_ALWAYS through Java reflection.