Skip to content

loadscope: forget a dead worker's collection so a still-collecting worker isn't mistaken for done - #1364

Open
dchaudhari7177 wants to merge 1 commit into
pytest-dev:masterfrom
dchaudhari7177:fix/loadscope-remove-node-registered-collections
Open

loadscope: forget a dead worker's collection so a still-collecting worker isn't mistaken for done#1364
dchaudhari7177 wants to merge 1 commit into
pytest-dev:masterfrom
dchaudhari7177:fix/loadscope-remove-node-registered-collections

Conversation

@dchaudhari7177

Copy link
Copy Markdown

Fixes #1362.

Problem

LoadScopeScheduling.remove_node() pops the dead worker from assigned_work but never removes its entry from registered_collections. Since collection_is_completed is len(registered_collections) >= numnodes, the stale entry keeps the collection looking "complete" while a later worker is still collecting. schedule() then runs and _assign_work_unit() raises KeyError indexing registered_collections for the worker that never registered — crashing the run with INTERNALERROR. This affects both --dist=loadscope and --dist=loadfile (the latter subclasses LoadScopeScheduling).

Fix

Drop the node from registered_collections in remove_node(), mirroring the assigned_work.pop(node) already there. A replacement worker then re-registers normally, and collection_is_completed correctly waits for every live worker.

This is safe for the shutdown path too: normal worker_workerfinished removals only happen after triggershutdown, and on a crash the replacement worker re-registers its collection.

Test

Added TestLoadScopeScheduling::test_remove_node_forgets_dead_worker_collection, which reproduces the scenario with MockNodes (3 expected workers; one dies mid-collection while another is still collecting). It fails on main (registered_collections still holds the dead node, collection_is_completed flips true early) and passes with the fix. Full testing/test_dsession.py is green; ruff check/format clean. Added changelog/1362.bugfix.rst.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

LoadScopeScheduling.remove_node leaves a dead worker in registered_collections, causing INTERNALERROR KeyError when another worker is still collecting

1 participant