In discussion with @plotnick: jobs currently are hard-killed if they exceed the (operator-defined) maximum stdout cap. This could be surprising, frustrating, or in some cases destructive, if the job is not, in some sense, "cancellation-safe", and the operator has misjudged (or forgotten to think about) how much output it will create.
Suggestion: it would be good if, instead of being kill -9'd, jobs which try to write past their quota block indefinitely, signaling back to the job manager that an "Output Too Big For He Gotdang File" event has occurred. By blocking the pipe into which the job writes, we apply backpressure to the job, which for most processes will stall them in some kind of reasonable way (because they, presumably, are meant to tolerate arbitrary I/O delays). This does not require SIGSTOP-ing the job, though we could do that too?
When such a pause occurs, we should set the status of the job to Paused, entering this back as a job-related event in the gossip network. Querying the status of the job, whether by polling or waiting for updates, should return to the client that the job has been paused. At this point, the client may make one of two decisions, the first of which is already wired in:
- Actually kill the job (we already have this capability).
- Allocate additional quota to the job as a per-job temporary override.
Either of these options, like all other job control commands, are broadcast back from the sush server into the gossip network, eventually reaching the sled which holds the paused job. If the client selects option 2, the job's max output size is increased, the concomitant backpressure is relieved, and it continues apace.
I don't think that much about this is difficult to wire up, because it operates through the same job control mechanisms that already exist. The most significant part, standing at a thousand feet away, is likely the backpressuring quota-enforcement pipe, which I suspect looks something like a triple of an AsyncRead, an AsyncWrite, and a watch::Receiver<usize>, which itself implements AsyncWrite, so that we can redirect output from the process into it.
In discussion with @plotnick: jobs currently are hard-killed if they exceed the (operator-defined) maximum stdout cap. This could be surprising, frustrating, or in some cases destructive, if the job is not, in some sense, "cancellation-safe", and the operator has misjudged (or forgotten to think about) how much output it will create.
Suggestion: it would be good if, instead of being
kill -9'd, jobs which try to write past their quota block indefinitely, signaling back to the job manager that an "Output Too Big For He Gotdang File" event has occurred. By blocking the pipe into which the job writes, we apply backpressure to the job, which for most processes will stall them in some kind of reasonable way (because they, presumably, are meant to tolerate arbitrary I/O delays). This does not require SIGSTOP-ing the job, though we could do that too?When such a pause occurs, we should set the status of the job to Paused, entering this back as a job-related event in the gossip network. Querying the status of the job, whether by polling or waiting for updates, should return to the client that the job has been paused. At this point, the client may make one of two decisions, the first of which is already wired in:
Either of these options, like all other job control commands, are broadcast back from the sush server into the gossip network, eventually reaching the sled which holds the paused job. If the client selects option 2, the job's max output size is increased, the concomitant backpressure is relieved, and it continues apace.
I don't think that much about this is difficult to wire up, because it operates through the same job control mechanisms that already exist. The most significant part, standing at a thousand feet away, is likely the backpressuring quota-enforcement pipe, which I suspect looks something like a triple of an
AsyncRead, anAsyncWrite, and awatch::Receiver<usize>, which itself implementsAsyncWrite, so that we can redirect output from the process into it.