-
Notifications
You must be signed in to change notification settings - Fork 694
Honor max_output_length for custom plugin stdout capture #1288
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -33,9 +33,18 @@ import ( | |
| "k8s.io/node-problem-detector/pkg/util/tomb" | ||
| ) | ||
|
|
||
| // maxCustomPluginBufferBytes is the max bytes that a custom plugin is allowed to | ||
| // send to stdout/stderr. Any bytes exceeding this value will be truncated. | ||
| const maxCustomPluginBufferBytes = 1024 * 4 | ||
| // maxCustomPluginStdoutCeilingBytes is a hard safety ceiling on how many bytes | ||
| // of stdout NPD reads from a custom plugin, bounding memory use from a runaway | ||
| // plugin. The per-plugin max_output_length is what actually governs the message | ||
| // size; this ceiling only caps the (misconfigured) case where max_output_length | ||
| // itself exceeds it. It must be >= the largest supported max_output_length so a | ||
| // plugin's configured limit is never silently truncated by the read buffer. | ||
| const maxCustomPluginStdoutCeilingBytes = 1024 * 1024 | ||
|
|
||
| // maxCustomPluginStderrBytes caps stderr. run() uses stderr only for diagnostic | ||
| // logging (logPluginStderr) and never includes it in the returned output, so it | ||
| // does not need to honor max_output_length and keeps a small fixed cap. | ||
| const maxCustomPluginStderrBytes = 1024 * 4 | ||
|
|
||
| type Plugin struct { | ||
| config cpmtypes.CustomPluginConfig | ||
|
|
@@ -210,14 +219,22 @@ func (p *Plugin) run(rule cpmtypes.CustomRule) (exitStatus cpmtypes.Status, outp | |
| stderrErr error | ||
| ) | ||
|
|
||
| // Capture enough stdout to honor the configured max_output_length, bounded | ||
| // by a hard safety ceiling. Previously this was a fixed 4 KiB buffer that | ||
| // silently truncated plugins configured with a larger max_output_length. | ||
| stdoutCapture := int64(*p.config.PluginGlobalConfig.MaxOutputLength) | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This changes the default config behavior. run() trims whitespace after capture so the old code always had a 4 KiB window to find non-whitespace content in. Now that window is max_output_length itself which is 80 bytes by default. I reproduced this is by having a plugin prints 80 spaces then Keeping the old 4 KiB as a floor fixes the >4 KiB truncation without changing anything else: stdoutCapture := int64(*p.config.PluginGlobalConfig.MaxOutputLength)
if stdoutCapture < 4096 {
stdoutCapture = 4096
}
if stdoutCapture > maxCustomPluginStdoutCeilingBytes {
stdoutCapture = maxCustomPluginStdoutCeilingBytes
}A leading whitespace test case that tests this is worthwhile |
||
| if stdoutCapture > maxCustomPluginStdoutCeilingBytes { | ||
| stdoutCapture = maxCustomPluginStdoutCeilingBytes | ||
| } | ||
|
|
||
| wg.Add(2) | ||
| go func() { | ||
| defer wg.Done() | ||
| stdout, stdoutErr = readFromReader(stdoutPipe, maxCustomPluginBufferBytes) | ||
| stdout, stdoutErr = readFromReader(stdoutPipe, stdoutCapture) | ||
| }() | ||
| go func() { | ||
| defer wg.Done() | ||
| stderr, stderrErr = readFromReader(stderrPipe, maxCustomPluginBufferBytes) | ||
| stderr, stderrErr = readFromReader(stderrPipe, maxCustomPluginStderrBytes) | ||
| }() | ||
| // This will wait for the reads to complete. If the execution times out, the pipes | ||
| // will be closed and the wait group unblocks. | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| #!/usr/bin/env bash | ||
|
|
||
| # Emits 16384 bytes of stdout (larger than the old hardcoded 4 KiB capture | ||
| # buffer and larger than the test's max_output_length) with a success exit | ||
| # code. Used to verify that plugin output capture honors max_output_length | ||
| # rather than a fixed internal buffer size. | ||
| head -c 16384 /dev/zero | tr '\0' 'a' | ||
| exit 0 |
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
max_output_length> 1 MiB gets capped here with no log. Would it be better to check it once at config load and log or reject out-of-range values there?1 mib may also be too big. The captured results still end up in
resultChanwhich holds 1000 entries. That is 1000 * 1 MiB =~ about 1 GiB of RAM held by a node daemon. And the output goes verbatim into NodeCondition.Message and event messages.