bug: Prevent exec hang when task decode fails

[coderfender]

Which issue does this PR close?

Closes #1908

Rationale for this change

What changes are included in this PR?

Previously, all errors were masked in a blanket error but we now handle task serialization / gRPC errors in a separate path and send an event to fail the corresponding job without marking the executor as dead essentially failing the query. This should help with cluster stability and determinism with regards to task serialization errors

Are there any user-facing changes?

[andygrove]

Thanks @coderfender. The changes look reasonable, but is it possible to add tests to prevent regressions?

[coderfender]

Added tests

[coderfender]

@phillipleblanc , thank you for the feedback. Please take a look whenever you can. One of the things I have observed while implementing review feedback and failing only invalid job(s) instead of all the jobs part of a batch RPC is to report the cores back t be reallocated as well. Now, the current logic assumes one core per task which should be true for most if not all cases . My plan is to create a followup on tackling the core count issue as a follow and keep this PR isolated to job scheduling aspects only

[phillipleblanc]

Thanks, this looks good to me now.

[milenkovicm]

hey @coderfender sorry for late review, i think this makes sense. I have one comment regarding return types of method. i find it a bit cunfusing that we return Ok and then list of failed jobs, could we find a better way to indicate partial failure, wdyt ?