ateapi: backfill unset worker state during pod sync - #599
Open
Mesut Oezdil (mesutoezdil) wants to merge 1 commit into
Open
ateapi: backfill unset worker state during pod sync#599Mesut Oezdil (mesutoezdil) wants to merge 1 commit into
Mesut Oezdil (mesutoezdil) wants to merge 1 commit into
Conversation
Mesut Oezdil (mesutoezdil)
force-pushed
the
fix/worker-state-backfill
branch
2 times, most recently
from
August 26, 2026 07:11
3bd62d1 to
8ddf9a1
Compare
Follow-up to agent-substrate#517, which added a scheduler filter that skips any worker whose state is not STATE_ACTIVE. The syncer only writes State when it creates a worker record; the update path in syncWorkerToStore diffs Ip, SandboxClass and Labels and never touches State, and no other code path writes it onto an existing record. The store is Redis backed, so worker records outlive an ate-api-server rollout, and worker pods are not recreated by that rollout either. Every worker record written before the State field existed therefore stays at STATE_UNSPECIFIED and is skipped by the scheduler for the whole life of its pod. On an existing cluster upgrading past agent-substrate#517, actor placement fails with ErrNoCapacity until each worker pod is deleted and recreated. The startup sync does not help, since the field diff computes changed == false and never writes.
Mesut Oezdil (mesutoezdil)
force-pushed
the
fix/worker-state-backfill
branch
from
August 26, 2026 15:39
8ddf9a1 to
a0a988b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #517.
#517 added a scheduler filter that skips any worker whose state is not
STATE_ACTIVE:substrate/cmd/ateapi/internal/scheduling/scheduling.go
Lines 113 to 115 in 87698ea
But the syncer writes
Stateonly when it creates a worker record. The update path insyncWorkerToStorediffsIp,SandboxClassandLabelsand never touchesState, and no other code path writesStateonto an existing record.The store is Redis backed, so worker records outlive an ate-api-server rollout, and worker pods are not recreated by that rollout either. Every worker record written before the
Statefield existed therefore stays atSTATE_UNSPECIFIEDand is skipped by the scheduler for the whole life of its pod. On an existing cluster upgrading past #517, actor placement fails withErrNoCapacityuntil each worker pod is deleted and recreated. The startup sync does not help, because the field diff computeschanged == falseand never writes.