fix: explain locality conflicts instead of 'no free workers available'#401
Open
Grant McCloskey (MushuEE) wants to merge 2 commits into
Open
fix: explain locality conflicts instead of 'no free workers available'#401Grant McCloskey (MushuEE) wants to merge 2 commits into
Grant McCloskey (MushuEE) wants to merge 2 commits into
Conversation
When FinalizePausedStep runs after the worker record is already gone, the node holding the local checkpoint is unknown. Previously the step wrote [""] into NodeVmsWithLocalSnapshots, which findFreeWorker treats as a hard restriction no worker can satisfy - every subsequent resume fails with 'no free workers available' and the actor is permanently unschedulable short of boot=true. Write side: only record the node when it is known; an empty list means no locality restriction, so the scheduler can still place the actor (local checkpoints live on the node, so a shuffled retry can land on the node that holds it). Read side: findFreeWorker ignores empty entries in the restriction list, healing actor records already poisoned by the old write path without a data migration. Fixes agent-substrate#397
When an actor's latest snapshot is a local checkpoint, findFreeWorker hard-restricts candidates to the node(s) holding it. When that restriction eliminates every candidate, the error was the generic 'no free workers available', which reads as cluster-wide capacity exhaustion - and hides the far more serious case where the snapshot's node no longer exists and the actor's checkpointed state is unreachable. findFreeWorker now returns a workerScan carrying the picked worker plus one flag gathered in its existing single pass (no extra iteration): whether any eligible worker exists on the restricted nodes. The failure message distinguishes: - all eligible workers on the snapshot's node(s) are busy - no eligible workers exist on the snapshot's node(s) at all (node likely removed) The status code remains FailedPrecondition; the generic message is unchanged when no locality restriction is in effect. Fixes agent-substrate#398
Collaborator
|
needs rebase |
Tim Hockin (thockin)
approved these changes
Jul 8, 2026
Tim Hockin (thockin)
left a comment
Tim Hockin (thockin)
left a comment
Collaborator
There was a problem hiding this comment.
Ping me after you rebase
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #398
Note
Based on #399 (both change
findFreeWorker); the first commit here is that PR. Review only the second commit, or merge #399 first and this rebases cleanly.When an actor's latest snapshot is a local checkpoint,
findFreeWorkerhard-restricts candidates to the node(s) holding it — correctly, since the checkpoint only exists there. But when that restriction eliminates every candidate, the error was the genericno free workers available, which reads as cluster-wide capacity exhaustion and hides the far more serious case where the snapshot's node no longer exists and the actor's checkpointed state is unreachable.findFreeWorkernow returns aworkerScancarrying the picked worker plus one flag gathered in its existing single pass (no extra iteration): whether any eligible worker exists on the restricted node(s). The failure message distinguishes:actor's local snapshot is on node(s) [X] but all eligible workers on those nodes are busyactor's local snapshot is on node(s) [X] but no eligible workers exist on those nodes (the node(s) may have been removed)The status code stays
FailedPrecondition, and the generic message is unchanged when no locality restriction is in effect (existing functional tests pass unmodified).This intentionally does not make the actor schedulable in either case — that requires the snapshot model to retain the external URI alongside local locality, which is part of a separate architecture proposal.