Worker: handle events getting rejected by lightning

[josephjclark]

It seems that if Lightning rejects an event like step:complete, the worker doesn't do anything with it.

We need to add some logic here.

What should we do?

• if we continue to send other events, lightning will just reject them
• so maybe better to send an error event and just abort the run - mark it with Exception

[josephjclark]

It might be tricky to make this work nicely in the worker.

I suppose on run:complete everything is done, and the error needs to be something like: "The workflow completed but failed to failed to send a final status to Lightning"

We also need an error for step:complete, and that bascally needs to shut down the run with a message like "your workflow might actually be carrying on, but lightning has closed it down". Which really isn't so helpful actually! So maybe we do need to force kill it at this point, if we can. For which I don't think we have a hook - which means this is a stealth abort signal in the runtime and engine.