[FLINK-38534][runtime/tests] Fix flaky LocalRecoveryTest

[mukul-8]

The test was flaky because it triggered a checkpoint immediately after calling setAllExecutionsToRunning() without verifying that tasks actually transitioned to RUNNING state. While state updates are synchronous, they trigger async callbacks that need time to process. In slower CI environments, the checkpoint coordinator may not have finished processing state updates, causing the checkpoint trigger to be rejected and the test to timeout.

Added waitForAllTasksRunning() utility method to ensure all task executions are in RUNNING state before triggering the checkpoint, eliminating the race condition.

Verified with 300+ local test iterations without failure.

What is the purpose of the change

Fixes FLINK-38534 - flaky LocalRecoveryTest.testStateSizeIsConsideredForLocalRecoveryOnRestart that intermittently times out in CI.

Root Cause: The test triggers a checkpoint immediately after setAllExecutionsToRunning() without verifying tasks have fully transitioned to RUNNING state. While state updates are
synchronous, they trigger async callbacks (checkpoint coordinator, etc.) that need time to process. In slower CI environments, the checkpoint coordinator may not have finished processing,
causing the checkpoint trigger to be rejected and waitForCheckpointInProgress() to timeout after 60 seconds.

Solution: Added waitForAllTasksRunning() utility method to ensure all task executions are in RUNNING state before triggering the checkpoint.

This follows a similar approach to #27740 (FLINK-39182) which fixed threading/timing issues in ExecutionGraphRestartTest.

Brief change log

• Added waitForAllTasksRunning(ExecutionGraph) to SchedulerTestingUtils
• Updated LocalRecoveryTest to wait for all tasks to reach RUNNING state before triggering checkpoint

Verifying this change

Verified locally with 300+ test iterations without failure.

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): no
• The public API, i.e., is any changed class annotated with @Public(Evolving): no
• The serializers: no
• The runtime per-record code paths (performance sensitive): no
• Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
• The S3 file system connector: no

Documentation

• Does this pull request introduce a new feature? no
• If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

[flinkbot]

CI report:

• 103d24d Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build

[davidradl]

We have seen other tests with this sort of resolution. I think this is reasonable. If we see timeouts then there will be another new underlying issue, casing the vertices not to get to running. Is the timeout long enough for the slow CIs?