Skip to content

Quiesce every Python worker path before deregistration #18

Description

@rmcdaniel

Problem

Graceful shutdown can remove a Python worker registration before every execution path has stopped using it.

run_until() performs its workflow and activity polling inline and does not observe an external stop() request. The shutdown path therefore has no task to cancel or await before deregistration. Separately, the dedicated query-poller thread is joined only for a bounded interval; shutdown does not verify that the thread exited before sending the deregistration request.

Either path can continue polling, handling, or completing work after the Server has removed the registration and recovered its workflow-task leases. That creates stale completions and duplicate execution risk during an otherwise orderly shutdown.

Required behavior

Deregister only after every poller has stopped accepting new tasks and all accepted work has either drained or been cancelled according to the configured shutdown deadline.

Acceptance criteria

  • An external stop() causes an active run_until() call to stop polling and finish; no describe, poll, or completion request is sent after deregistration.
  • A blocked query long-poll is interrupted or fully joined before deregistration, and a query returned during shutdown is not dispatched after the registration is removed.
  • Inline run_until() workflow/activity handling is accounted for as in-flight work and follows the same drain deadline as run() dispatch.
  • If a poller cannot be quiesced, shutdown surfaces an actionable failure instead of reporting a clean stop while deregistering underneath live work.
  • Deterministic async/threaded tests hold each path open, call stop(), and prove request ordering around the single deregistration request.
  • Existing idempotent repeated-stop and primary-error preservation behavior remains intact.

Delete when

Delete when main and a published Python SDK prerelease prove that run(), run_until(), and query-thread shutdown all quiesce before worker deregistration under normal, timeout, and failure paths.

Metadata

Metadata

Assignees

No one assigned

    Labels

    authority:githubGitHub is the authoritative lifecycle record for this workbeta:compatibleMay be completed without changing the frozen beta contractkind:defectA public product behavior is incorrectpriority:P1High-priority product or release riskrepo:sdk-pythonOwned by the Python SDK repositorystatus:readyReady for implementation

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions