e2e: give each probe fixture its own worker pool - #1147
Open
NekoPunch (orangeCatDeveloper) wants to merge 1 commit into
Open
e2e: give each probe fixture its own worker pool#1147NekoPunch (orangeCatDeveloper) wants to merge 1 commit into
NekoPunch (orangeCatDeveloper) wants to merge 1 commit into
Conversation
The pool label and workerSelector were the constant workload: probe, so identity, egressmitm and imagevolume all selected the same workers. A suite tearing down its namespace deleted workers another suite's actors were still running on.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1146
TestActorIdentity_AfterRestore_IsOwnID_NotGoldenis the most frequent failure in CI — 26 of the last 80 failed runs — and fails three different ways: a dial to a worker'sateom.sockthat is not there,runsc restore`: signal: killed, and a router 502/503.All three are one bug.
probe.yaml.tmplgives every suite its own namespace, ActorTemplate and snapshot prefix through${FIXTURE_SUFFIX}, but the pool label and theworkerSelectorare the constantworkload: probe. Selection is by label, not namespace, soidentity,egressmitmandimagevolumeshare one 9-worker pool, and the first suite to tear its namespace down deletes workers the others are still running actors on. The victim's symptom is decided only by where it happened to be: between calls it cannot dial, mid-restore ateom's handler context is cancelled andexec.CommandContextSIGKILLs runsc, and after routing is up the router answers 5xx.Putting the suffix on both labels restores the isolation the rest of the template already has —
parking_test.go:190does the same withdemo: <namespace>.capabilities.yaml.tmplandprobe-sized.yaml.tmplcarry constant labels too, but each has a single caller today, so they cannot collide and are left alone.Two identical 4-vCPU VMs, run concurrently, differing only in whether these two labels carry the suffix. A round is the full e2e suite; a hit is any of the three signatures:
The unfixed arm also carries CI's own evidence — an identity actor running on an imagevolume worker: