Skip to content

e2e: give each probe fixture its own worker pool - #1147

Open
NekoPunch (orangeCatDeveloper) wants to merge 1 commit into
agent-substrate:mainfrom
orangeCatDeveloper:fix/e2e-probe-pool-isolation
Open

e2e: give each probe fixture its own worker pool#1147
NekoPunch (orangeCatDeveloper) wants to merge 1 commit into
agent-substrate:mainfrom
orangeCatDeveloper:fix/e2e-probe-pool-isolation

Conversation

@orangeCatDeveloper

@orangeCatDeveloper NekoPunch (orangeCatDeveloper) commented Aug 22, 2026

Copy link
Copy Markdown
Contributor

Fixes #1146

TestActorIdentity_AfterRestore_IsOwnID_NotGolden is the most frequent failure in CI — 26 of the last 80 failed runs — and fails three different ways: a dial to a worker's ateom.sock that is not there, runsc restore`: signal: killed, and a router 502/503.

All three are one bug. probe.yaml.tmpl gives every suite its own namespace, ActorTemplate and snapshot prefix through ${FIXTURE_SUFFIX}, but the pool label and the workerSelector are the constant workload: probe. Selection is by label, not namespace, so identity, egressmitm and imagevolume share one 9-worker pool, and the first suite to tear its namespace down deletes workers the others are still running actors on. The victim's symptom is decided only by where it happened to be: between calls it cannot dial, mid-restore ateom's handler context is cancelled and exec.CommandContext SIGKILLs runsc, and after routing is up the router answers 5xx.

Putting the suffix on both labels restores the isolation the rest of the template already has — parking_test.go:190 does the same with demo: <namespace>.

capabilities.yaml.tmpl and probe-sized.yaml.tmpl carry constant labels too, but each has a single caller today, so they cannot collide and are left alone.

Two identical 4-vCPU VMs, run concurrently, differing only in whether these two labels carry the suffix. A round is the full e2e suite; a hit is any of the three signatures:

unfixed   114 rounds   28 hits    dial 15   runsc-killed 6   router 5xx 7
fixed      92 rounds    0 hits    dial  0   runsc-killed 0   router 5xx 0

Fisher two-sided, any signature: p = 1.2e-8

The unfixed arm also carries CI's own evidence — an identity actor running on an imagevolume worker:

runs/32548404077
  identity_test.go:164: SuspendActor "probe-alpha": rpc error: code = Unavailable
    desc = "transport: Error while dialing: dial unix /var/lib/ateom-gvisor/ateoms/63f963d9-…/ateom.sock"
  ateapi: "Syncer: removing worker from store (pod deleted)"
    worker=63f963d9-…  pod=ate-e2e-probe-imagevolume/probe-6cfd69bbbd-xl29r
  • Tests pass
  • Appropriate changes to documentation are included in the PR

The pool label and workerSelector were the constant workload: probe, so
identity, egressmitm and imagevolume all selected the same workers. A
suite tearing down its namespace deleted workers another suite's actors
were still running on.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

E2E test flake: TestActorIdentity_AfterRestore_IsOwnID_NotGolden — three suites share one worker pool

1 participant