make a killed runsc say who killed it - #1154
Open
NekoPunch (orangeCatDeveloper) wants to merge 1 commit into
Open
make a killed runsc say who killed it#1154NekoPunch (orangeCatDeveloper) wants to merge 1 commit into
NekoPunch (orangeCatDeveloper) wants to merge 1 commit into
Conversation
CI dumps tailed every pod at 300 lines and never looked at termination state, node conditions, events or dmesg, so a failure whose explanation had scrolled off could not be diagnosed from the run. And runsc reports `signal: killed` for an OOM kill, an outside SIGKILL and a cancelled context alike, with nothing in the error to tell them apart.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1153
A CI run where runsc was killed says neither why it died nor enough to work it out afterwards.
runscreportssignal: killedfor an OOM kill, an outside SIGKILL andexec.CommandContextacting on a done context alike, and every subcommand inrunsc.godroppedctx.Err(), the one thing that separates them. Meanwhile the dump tailed each pod at 300 lines and never looked at termination state, node conditions, events or dmesg, so the line that would have explained a failure had usually scrolled off.ctxErrorSuffixnames the context state next to the failed command, and stays silent when the error already carries it — which is whatRunreturns if the context died beforeStart. It says what was observed rather than naming a killer: a done context makes cancellation the strong candidate, but the process could still have been killed for another reason at the same moment.On the CI side the workers now dump untailed, plus their previous container's log and
describe, while the control plane keeps its tail: it runs the length of the job, and its history is not where a failing actor's evidence is. dmesg is grepped for OOM rather than tailed, since the kill line scrolls off a busy ring buffer, with a plain tail kept alongside for anything the grep misses.Reading the two together: a context error points at cancellation, a dmesg record confirms an OOM, and neither means the cause is still open.
Before and after, for the three ways the subprocess dies:
The dump was run against a live cluster: every section produced output, and the OOM grep found a real record a
tail -100would have missed.hack/verify-all.shpasses exceptproto-fmt, which needs a clang-format this machine does not have; no proto files are touched.