integration multikueue tests suite is flaking

[mimowo]

Which test is flaking?:
don't know
First observed in (PR or commit, if known):
no idea
Link to failed CI job or steps to reproduce locally:
https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/periodic-kueue-test-integration-multikueue-release-0-14/2009115078468571136
Failure message or logs:

/home/prow/go/src/kubernetes-sigs/kueue/test/integration/framework/framework.go:153
------------------------------
[DeferCleanup (Suite)] PASSED [0.001 seconds]
[DeferCleanup (Suite)] 
/home/prow/go/src/kubernetes-sigs/kueue/test/integration/framework/framework.go:153
------------------------------
[DeferCleanup (Suite)] PASSED [0.001 seconds]
[DeferCleanup (Suite)] 
/home/prow/go/src/kubernetes-sigs/kueue/test/integration/framework/framework.go:153
------------------------------
[ReportAfterSuite] PASSED [3.409 seconds]
[ReportAfterSuite] Autogenerated ReportAfterSuite for --json-report --junit-report
autogenerated by Ginkgo
------------------------------
Ran 36 of 36 Specs in 185.906 seconds
SUCCESS! -- 36 Passed | 0 Failed | 0 Pending | 0 Skipped
Ginkgo ran 2 suites in 5m7.28213216s
There were failures detected in the following suites:
  multikueue ./test/integration/multikueue
Test Suite Failed
make: *** [Makefile-test.mk:90: test-multikueue-integration] Error 1

Anything else we need to know?:

[mimowo]

cc @mszadkow @mbobrovskyi who may have investigated similar issues with the integration tests in the past (no test errors, but suite failing)

[mimowo]

/priority important-soon

[skools-here]

I’ve looked into this and the flakiness seems to come from aggressive shared timeouts in the multikueue integration test utilities. In CI, controller startup and cross-cluster reconciliation can exceed the current limits. I’m planning to propose a fix by increasing the common test timeouts to make the suite deterministic. I’ll open a PR shortly.

/assign

[mimowo]

@skools-here this looks promising. If this is indeed due to aggressive timeout, then we should be able to confirm the hypothesis by lowering the timeout and seeing if we can trigger the failure more reliably, wdyt?

[mbobrovskyi]

I think this is a race condition. We already hit the same issue with @mykysha and @mszadkow (#8070). The problem is that --output-interceptor-mode=none doesn’t allow us to see race condition logs.

[mbobrovskyi]

I’ve looked into this and the flakiness seems to come from aggressive shared timeouts in the multikueue integration test utilities. In CI, controller startup and cross-cluster reconciliation can exceed the current limits. I’m planning to propose a fix by increasing the common test timeouts to make the suite deterministic. I’ll open a PR shortly.

I don’t think this is the problem. Timeout issues should show us the exact error in the test case.

[mbobrovskyi]

/cc @mszadkow

[mszadkow]

Yes, I do think so too.
Unfortunately there is no good solution for that, we can either pass --output-interceptor-mode=none or not and both are not optimal.
Related to #8070