test(e2e): fix flaky e2e tests - #4443
Open
stevenvegt wants to merge 1 commit into
Open
Conversation
assertJaegerTrace only retried while the trace was entirely absent from Jaeger. Each node exports spans in batches on its own schedule (5s default), so the first query after a flow often finds a partial trace containing only one node's spans, which failed the assertion immediately. Missing services or spans are now retried like a missing trace, up to 10 attempts with 1s sleeps. The gossip test gave propagation of 81 transactions across 4 nodes only 10 seconds, the heaviest propagation assertion in the suite on the same budget as tests waiting for at most 41 transactions. Slow CI runners regularly miss that deadline; it is now 30 seconds. Both were observed as e2e failures on unrelated PRs (#4427, #4439). Assisted-by: AI
stevenvegt
requested review from
Dirklectisch,
JorisHeadease,
gerardsn,
reinkrul and
woutslakhorst
as code owners
July 30, 2026 15:37
Contributor
|
Coverage Impact This PR will not change total coverage. 🚦 See full report on Qlty Cloud »🛟 Help
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Fixes two e2e flakes that recently failed CI on unrelated PRs (#4427, #4439).
Jaeger trace assertion (
util.sh):assertJaegerTraceonly retried while the trace was entirely absent from Jaeger. Each node exports spans in batches on its own schedule (5s default interval), so the first query after a flow can find a partial trace containing only one node's spans, which failed the assertion immediately (observed on #4427:FAILED: Trace '...' missing service 'nodeA' (found: nodeB), 30ms after the check started, while two earlier rfc021 variants passed the same check). Missing services or spans are now retried like a missing trace, up to 10 attempts with 1s sleeps, which comfortably covers the batch export interval. The success path is unchanged, and the final error message still reports what was missing on the last attempt.Gossip propagation deadline (
nuts-network/gossip/run-test.sh): the test waits for 81 transactions to propagate across 4 nodes but gave it only 10 seconds, the heaviest propagation assertion in the suite on the same budget as tests waiting for at most 41 transactions. Slow CI runners regularly miss that deadline (observed on #4439). The deadline is now 30 seconds;waitForTXCountpolls every second, so passing runs are not slowed down.To be backported to release branches later, since PR CI on those branches runs the
e2e-tests/directory from their own merge ref.