Skip to content

Python: FHA LRA workflow checkpoint gap investigation #7809

Description

@TaoChenOSU

In FHA LRA with MAF workflows, there are a few known gaps that require more in-depth investigation:

  1. A workflow that doesn't produce any output until the very end will never be recover from a crash, because a responses checkpoint can only be persisted when the workflow yields output. Upon recovery, the workflow will be restarted from the very beginning, which is not desired.
  2. There is always a gap between when the workflow stores its checkpoint and when the responses checkpoint is persisted. Say a superstep that yields two outputs, the first output gets to the client while the second output is still on the wire, the connection drops (client never gets the second output) and the workflow moves the next superstep after storing a checkpoint, and now the process crashes, the checkpoint gets stored but is never persisted with the response stream so upon recovery the workflow resumes from the checkpoint, which will not rerun the previous superstep. The client will lose the second update forever.

Metadata

Metadata

Labels

foundryUsage: [Issues, PRs], Target: all Foundry integrationshostingUsage: [Issues, PRs], Target: all hosting related solutionspythonUsage: [Issues, PRs], Target: Python

Type

Projects

Status
No status

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions