Unset inherited JVM env vars in crashtracking scripts#10819
Draft
Unset inherited JVM env vars in crashtracking scripts#10819
Conversation
The OOME notifier and crash uploader scripts now unset JDK_JAVA_OPTIONS, JAVA_TOOL_OPTIONS, and _JAVA_OPTIONS before spawning a child JVM. This prevents port conflicts, memory contention, and lost diagnostics when the parent environment contains flags like JMX remote ports. Fixes #10766 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
BenchmarksStartupParameters
See matching parameters
SummaryFound 1 performance improvements and 0 performance regressions! Performance is the same for 64 metrics, 6 unstable metrics.
Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.61.0-SNAPSHOT~364ba98ac2, baseline=1.61.0-SNAPSHOT~fd65c0aa59
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.078 s) : 0, 1078258
Total [baseline] (8.908 s) : 0, 8907611
Agent [candidate] (1.073 s) : 0, 1072587
Total [candidate] (8.876 s) : 0, 8876207
section iast
Agent [baseline] (1.226 s) : 0, 1225957
Total [baseline] (9.559 s) : 0, 9559021
Agent [candidate] (1.227 s) : 0, 1227310
Total [candidate] (9.553 s) : 0, 9552878
gantt
title insecure-bank - break down per module: candidate=1.61.0-SNAPSHOT~364ba98ac2, baseline=1.61.0-SNAPSHOT~fd65c0aa59
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.229 ms) : 0, 1229
crashtracking [candidate] (1.204 ms) : 0, 1204
BytebuddyAgent [baseline] (640.092 ms) : 0, 640092
BytebuddyAgent [candidate] (637.242 ms) : 0, 637242
AgentMeter [baseline] (29.619 ms) : 0, 29619
AgentMeter [candidate] (29.531 ms) : 0, 29531
GlobalTracer [baseline] (260.62 ms) : 0, 260620
GlobalTracer [candidate] (260.024 ms) : 0, 260024
AppSec [baseline] (32.281 ms) : 0, 32281
AppSec [candidate] (32.069 ms) : 0, 32069
Debugger [baseline] (59.807 ms) : 0, 59807
Debugger [candidate] (59.691 ms) : 0, 59691
Remote Config [baseline] (636.127 µs) : 0, 636
Remote Config [candidate] (616.341 µs) : 0, 616
Telemetry [baseline] (8.833 ms) : 0, 8833
Telemetry [candidate] (8.767 ms) : 0, 8767
Flare Poller [baseline] (8.819 ms) : 0, 8819
Flare Poller [candidate] (7.191 ms) : 0, 7191
section iast
crashtracking [baseline] (1.19 ms) : 0, 1190
crashtracking [candidate] (1.197 ms) : 0, 1197
BytebuddyAgent [baseline] (795.91 ms) : 0, 795910
BytebuddyAgent [candidate] (796.873 ms) : 0, 796873
AgentMeter [baseline] (11.316 ms) : 0, 11316
AgentMeter [candidate] (11.341 ms) : 0, 11341
GlobalTracer [baseline] (247.465 ms) : 0, 247465
GlobalTracer [candidate] (247.594 ms) : 0, 247594
AppSec [baseline] (26.424 ms) : 0, 26424
AppSec [candidate] (26.329 ms) : 0, 26329
Debugger [baseline] (62.409 ms) : 0, 62409
Debugger [candidate] (62.609 ms) : 0, 62609
Remote Config [baseline] (512.386 µs) : 0, 512
Remote Config [candidate] (496.796 µs) : 0, 497
Telemetry [baseline] (14.771 ms) : 0, 14771
Telemetry [candidate] (14.696 ms) : 0, 14696
Flare Poller [baseline] (4.832 ms) : 0, 4832
Flare Poller [candidate] (4.91 ms) : 0, 4910
IAST [baseline] (25.137 ms) : 0, 25137
IAST [candidate] (25.177 ms) : 0, 25177
Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.61.0-SNAPSHOT~364ba98ac2, baseline=1.61.0-SNAPSHOT~fd65c0aa59
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.068 s) : 0, 1067874
Total [baseline] (11.039 s) : 0, 11038551
Agent [candidate] (1.059 s) : 0, 1059491
Total [candidate] (11.12 s) : 0, 11120084
section appsec
Agent [baseline] (1.252 s) : 0, 1252191
Total [baseline] (11.148 s) : 0, 11147702
Agent [candidate] (1.248 s) : 0, 1248169
Total [candidate] (11.172 s) : 0, 11172370
section iast
Agent [baseline] (1.244 s) : 0, 1244489
Total [baseline] (11.348 s) : 0, 11347514
Agent [candidate] (1.229 s) : 0, 1228607
Total [candidate] (11.361 s) : 0, 11360511
section profiling
Agent [baseline] (1.194 s) : 0, 1193714
Total [baseline] (11.081 s) : 0, 11080737
Agent [candidate] (1.181 s) : 0, 1180845
Total [candidate] (11.072 s) : 0, 11071567
gantt
title petclinic - break down per module: candidate=1.61.0-SNAPSHOT~364ba98ac2, baseline=1.61.0-SNAPSHOT~fd65c0aa59
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.199 ms) : 0, 1199
crashtracking [candidate] (1.186 ms) : 0, 1186
BytebuddyAgent [baseline] (632.627 ms) : 0, 632627
BytebuddyAgent [candidate] (629.57 ms) : 0, 629570
AgentMeter [baseline] (29.278 ms) : 0, 29278
AgentMeter [candidate] (29.156 ms) : 0, 29156
GlobalTracer [baseline] (258.23 ms) : 0, 258230
GlobalTracer [candidate] (257.591 ms) : 0, 257591
AppSec [baseline] (31.756 ms) : 0, 31756
AppSec [candidate] (31.48 ms) : 0, 31480
Debugger [baseline] (59.845 ms) : 0, 59845
Debugger [candidate] (59.368 ms) : 0, 59368
Remote Config [baseline] (612.99 µs) : 0, 613
Remote Config [candidate] (612.291 µs) : 0, 612
Telemetry [baseline] (8.699 ms) : 0, 8699
Telemetry [candidate] (8.668 ms) : 0, 8668
Flare Poller [baseline] (9.507 ms) : 0, 9507
Flare Poller [candidate] (5.853 ms) : 0, 5853
section appsec
crashtracking [baseline] (1.206 ms) : 0, 1206
crashtracking [candidate] (1.217 ms) : 0, 1217
BytebuddyAgent [baseline] (662.467 ms) : 0, 662467
BytebuddyAgent [candidate] (658.918 ms) : 0, 658918
AgentMeter [baseline] (12.114 ms) : 0, 12114
AgentMeter [candidate] (12.063 ms) : 0, 12063
GlobalTracer [baseline] (259.173 ms) : 0, 259173
GlobalTracer [candidate] (258.82 ms) : 0, 258820
IAST [baseline] (24.129 ms) : 0, 24129
IAST [candidate] (24.067 ms) : 0, 24067
AppSec [baseline] (177.557 ms) : 0, 177557
AppSec [candidate] (177.549 ms) : 0, 177549
Debugger [baseline] (65.982 ms) : 0, 65982
Debugger [candidate] (66.05 ms) : 0, 66050
Remote Config [baseline] (574.414 µs) : 0, 574
Remote Config [candidate] (567.122 µs) : 0, 567
Telemetry [baseline] (8.936 ms) : 0, 8936
Telemetry [candidate] (8.977 ms) : 0, 8977
Flare Poller [baseline] (3.612 ms) : 0, 3612
Flare Poller [candidate] (3.649 ms) : 0, 3649
section iast
crashtracking [baseline] (1.202 ms) : 0, 1202
crashtracking [candidate] (1.189 ms) : 0, 1189
BytebuddyAgent [baseline] (808.531 ms) : 0, 808531
BytebuddyAgent [candidate] (796.851 ms) : 0, 796851
AgentMeter [baseline] (11.747 ms) : 0, 11747
AgentMeter [candidate] (11.352 ms) : 0, 11352
GlobalTracer [baseline] (250.163 ms) : 0, 250163
GlobalTracer [candidate] (248.049 ms) : 0, 248049
IAST [baseline] (25.583 ms) : 0, 25583
IAST [candidate] (25.194 ms) : 0, 25194
AppSec [baseline] (26.884 ms) : 0, 26884
AppSec [candidate] (26.505 ms) : 0, 26505
Debugger [baseline] (63.634 ms) : 0, 63634
Debugger [candidate] (65.183 ms) : 0, 65183
Remote Config [baseline] (517.515 µs) : 0, 518
Remote Config [candidate] (519.768 µs) : 0, 520
Telemetry [baseline] (14.909 ms) : 0, 14909
Telemetry [candidate] (13.263 ms) : 0, 13263
Flare Poller [baseline] (4.871 ms) : 0, 4871
Flare Poller [candidate] (4.459 ms) : 0, 4459
section profiling
crashtracking [baseline] (1.185 ms) : 0, 1185
crashtracking [candidate] (1.169 ms) : 0, 1169
BytebuddyAgent [baseline] (689.714 ms) : 0, 689714
BytebuddyAgent [candidate] (682.199 ms) : 0, 682199
AgentMeter [baseline] (8.636 ms) : 0, 8636
AgentMeter [candidate] (8.607 ms) : 0, 8607
GlobalTracer [baseline] (217.027 ms) : 0, 217027
GlobalTracer [candidate] (215.251 ms) : 0, 215251
AppSec [baseline] (32.27 ms) : 0, 32270
AppSec [candidate] (31.882 ms) : 0, 31882
Debugger [baseline] (64.935 ms) : 0, 64935
Debugger [candidate] (65.217 ms) : 0, 65217
Remote Config [baseline] (600.595 µs) : 0, 601
Remote Config [candidate] (570.233 µs) : 0, 570
Telemetry [baseline] (9.084 ms) : 0, 9084
Telemetry [candidate] (8.147 ms) : 0, 8147
Flare Poller [baseline] (3.55 ms) : 0, 3550
Flare Poller [candidate] (3.51 ms) : 0, 3510
ProfilingAgent [baseline] (95.17 ms) : 0, 95170
ProfilingAgent [candidate] (93.553 ms) : 0, 93553
Profiling [baseline] (95.75 ms) : 0, 95750
Profiling [candidate] (94.112 ms) : 0, 94112
LoadParameters
See matching parameters
SummaryFound 1 performance improvements and 0 performance regressions! Performance is the same for 20 metrics, 15 unstable metrics.
Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.61.0-SNAPSHOT~364ba98ac2, baseline=1.61.0-SNAPSHOT~fd65c0aa59
dateFormat X
axisFormat %s
section baseline
no_agent (19.531 ms) : 19327, 19735
. : milestone, 19531,
appsec (18.333 ms) : 18146, 18519
. : milestone, 18333,
code_origins (17.947 ms) : 17766, 18128
. : milestone, 17947,
iast (17.947 ms) : 17767, 18126
. : milestone, 17947,
profiling (19.757 ms) : 19561, 19953
. : milestone, 19757,
tracing (17.718 ms) : 17542, 17895
. : milestone, 17718,
section candidate
no_agent (18.841 ms) : 18649, 19032
. : milestone, 18841,
appsec (18.632 ms) : 18443, 18820
. : milestone, 18632,
code_origins (17.797 ms) : 17618, 17976
. : milestone, 17797,
iast (17.786 ms) : 17610, 17962
. : milestone, 17786,
profiling (18.717 ms) : 18531, 18904
. : milestone, 18717,
tracing (17.611 ms) : 17437, 17785
. : milestone, 17611,
Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.61.0-SNAPSHOT~364ba98ac2, baseline=1.61.0-SNAPSHOT~fd65c0aa59
dateFormat X
axisFormat %s
section baseline
no_agent (1.191 ms) : 1180, 1203
. : milestone, 1191,
iast (3.205 ms) : 3162, 3247
. : milestone, 3205,
iast_FULL (5.664 ms) : 5609, 5720
. : milestone, 5664,
iast_GLOBAL (3.603 ms) : 3549, 3657
. : milestone, 3603,
profiling (2.101 ms) : 2081, 2120
. : milestone, 2101,
tracing (1.815 ms) : 1798, 1832
. : milestone, 1815,
section candidate
no_agent (1.181 ms) : 1169, 1192
. : milestone, 1181,
iast (3.184 ms) : 3142, 3227
. : milestone, 3184,
iast_FULL (5.656 ms) : 5599, 5712
. : milestone, 5656,
iast_GLOBAL (3.66 ms) : 3602, 3718
. : milestone, 3660,
profiling (2.06 ms) : 2042, 2078
. : milestone, 2060,
tracing (1.761 ms) : 1747, 1775
. : milestone, 1761,
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics. Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.61.0-SNAPSHOT~364ba98ac2, baseline=1.61.0-SNAPSHOT~fd65c0aa59
dateFormat X
axisFormat %s
section baseline
no_agent (1.463 ms) : 1451, 1474
. : milestone, 1463,
appsec (2.503 ms) : 2448, 2558
. : milestone, 2503,
iast (2.245 ms) : 2176, 2313
. : milestone, 2245,
iast_GLOBAL (2.287 ms) : 2217, 2356
. : milestone, 2287,
profiling (2.099 ms) : 2043, 2156
. : milestone, 2099,
tracing (2.048 ms) : 1994, 2101
. : milestone, 2048,
section candidate
no_agent (1.466 ms) : 1454, 1477
. : milestone, 1466,
appsec (2.559 ms) : 2501, 2616
. : milestone, 2559,
iast (2.242 ms) : 2173, 2311
. : milestone, 2242,
iast_GLOBAL (2.285 ms) : 2216, 2354
. : milestone, 2285,
profiling (2.087 ms) : 2030, 2143
. : milestone, 2087,
tracing (2.046 ms) : 1993, 2100
. : milestone, 2046,
Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.61.0-SNAPSHOT~364ba98ac2, baseline=1.61.0-SNAPSHOT~fd65c0aa59
dateFormat X
axisFormat %s
section baseline
no_agent (15.529 s) : 15529000, 15529000
. : milestone, 15529000,
appsec (15.291 s) : 15291000, 15291000
. : milestone, 15291000,
iast (18.493 s) : 18493000, 18493000
. : milestone, 18493000,
iast_GLOBAL (17.764 s) : 17764000, 17764000
. : milestone, 17764000,
profiling (15.124 s) : 15124000, 15124000
. : milestone, 15124000,
tracing (15.494 s) : 15494000, 15494000
. : milestone, 15494000,
section candidate
no_agent (15.305 s) : 15305000, 15305000
. : milestone, 15305000,
appsec (14.651 s) : 14651000, 14651000
. : milestone, 14651000,
iast (18.042 s) : 18042000, 18042000
. : milestone, 18042000,
iast_GLOBAL (17.871 s) : 17871000, 17871000
. : milestone, 17871000,
profiling (15.532 s) : 15532000, 15532000
. : milestone, 15532000,
tracing (15.281 s) : 15281000, 15281000
. : milestone, 15281000,
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What Does This Do
Clears
JDK_JAVA_OPTIONS,JAVA_TOOL_OPTIONS, and_JAVA_OPTIONSenvironment variables before spawning a child JVM in the OOME notifier and crash uploader scripts (both.shand.batvariants). This prevents the child JVM from inheriting the parent's environment, which can cause port conflicts, memory contention, and lost diagnostics.Motivation
When the Datadog admission controller injects the agent via
JAVA_TOOL_OPTIONS, and the application sets JMX or memory flags viaJDK_JAVA_OPTIONS, the child JVM spawned bydd_oome_notifier.sh/dd_crash_uploader.shinherits these variables. This causes failures such asBindExceptionon JMX ports still held by the parent, or cgroup OOM kills from competing memory reservations — resulting in lost OOME/crash events.GH Ticket #10766
Additional Notes
notify_oome.sh,notify_oome.bat,upload_crash.sh,upload_crash.batupload_crash.*are coveredJDK_JAVA_OPTIONSwith JMX port-binding flags and asserting events are still receivedContributor Checklist
type:and (comp:orinst:) labels in addition to any other useful labelsclose,fix, or any linking keywords when referencing an issueUse
solvesinstead, and assign the PR milestone to the issueNote: Once your PR is ready to merge, add it to the merge queue by commenting
/merge./merge -ccancels the queue request./merge -f --reason "reason"skips all merge queue checks; please use this judiciously, as some checks do not run at the PR-level. For more information, see this doc.