[CRE] confidential relay: wait for late execution handler before failing relay callback - #23284
Open
prashantkumar1982 wants to merge 2 commits into
Open
[CRE] confidential relay: wait for late execution handler before failing relay callback#23284prashantkumar1982 wants to merge 2 commits into
prashantkumar1982 wants to merge 2 commits into
Conversation
…ing relay callback The enclave runs one DON-shared confidential execution and only needs a relay quorum to proceed, so the gateway's relay callback (fanned out to every workflow-DON member) can reach a node before that node has started its own copy of the execution and registered the execution handler. The node then failed the callback with "execution handler not found", dropping its signature and, when enough nodes raced, pushing the relay below quorum. Add ExecutionHandlers.GetExecutionWithWait: on a miss it registers a lost-wakeup-safe waiter and blocks until the handler appears or the caller's context is done. The relay handler calls it with a bounded (5s) deadline, well under the gateway's relay request timeout, and only after attestation and Workflow-DON authorization pass so an unverified callback cannot park a waiter. This addresses the start-edge race; the end-edge (early RemoveExecution on execution timeout) and pathological never-starts (concurrency saturation, stale-event drops) are separate. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
|
👋 prashantkumar1982, thanks for creating this pull request! To help reviewers, please consider creating future PRs as drafts first. This allows you to self-review and make any final changes before notifying the team. Once you're ready, you can mark it as "Ready for review" to request feedback. Thanks! |
Contributor
|
✅ No conflicts with other open PRs targeting |
Satisfy golangci paralleltest, which flagged the new test functions. Also capture start before setting the deadline in the timeout test so the elapsed lower-bound assertion cannot flake. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Nodes were intermittently failing the enclave's confidential-relay callbacks with
execution handler for workflow <id> execution <id> not found, sometimes oftenenough that the gateway relay handler timed out without quorum and the whole
confidential execution failed.
Root cause is a start-edge race, not an early removal:
enclave only needs a relay quorum (
F+1..2F+1) to proceed.member (
fanOutToNodesranges overdonConfig.Members), not just the nodescurrently executing.
AddExecutioninsideConfidentialModule.Execute, which runs well after the engine'sWorkflow execution startinglog.So a node that is slightly behind (still scheduling the trigger, briefly blocked
on the per-workflow execution-concurrency limiter, etc.) can receive the relay
callback before it has started its own copy of the execution and registered a
handler. It then fails the callback instantly, dropping its signature. Confirmed
in the field: erroring nodes had no
Workflow execution startinglog for theexecution the gateway was asking about.
What this changes
Make the node-side relay handler tolerant of a not-yet-registered execution:
ExecutionHandlers.GetExecutionWithWait(ctx, workflowID, execID)— on a miss itregisters a waiter and blocks until
AddExecutionregisters the handler orctxis done. The check-then-register happens under a single lock hold thatAddExecutionalso takes, so there is no lost-wakeup window. Backed by amutex-guarded map + waiter registry (replacing the bare
sync.Map); the zerovalue is still ready to use.
handleSecretsGet/handleCapabilityExecutecall it with a bounded deadline(
defaultGetExecutionWait = 5s, well under the gateway's 30s relay requesttimeout so the node still answers in time to be counted), and — importantly —
only after attestation and Workflow-DON authorization pass, so an
unverified callback cannot make a node park a waiter. A callback for an
execution this node never runs still fails promptly once the bound elapses.