-
Notifications
You must be signed in to change notification settings - Fork 324
Fix DSM queue names with Kafka Connect IBM MQ connectors #10318
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix DSM queue names with Kafka Connect IBM MQ connectors #10318
Conversation
When using Kafka Connect with IBM MQ connectors, DSM was reporting incorrect queue names with schema-derived suffixes like _messagebody_0. This occurred because Kafka Connect schema converters add index suffixes to field names for union/optional types. This fix sanitizes queue/topic names to remove these suffixes: - _messagebody_N - _text_N - _bytes_N - _map_N - _value_N Fixes Zendesk ticket #2429181 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
BenchmarksStartupParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 60 metrics, 5 unstable metrics. Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.59.0-SNAPSHOT~078e8d1842, baseline=1.59.0-SNAPSHOT~372ceb0ed6
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.092 s) : 0, 1092286
Total [baseline] (8.728 s) : 0, 8727872
Agent [candidate] (1.086 s) : 0, 1085633
Total [candidate] (8.751 s) : 0, 8751215
section iast
Agent [baseline] (1.224 s) : 0, 1224475
Total [baseline] (9.292 s) : 0, 9292369
Agent [candidate] (1.226 s) : 0, 1225934
Total [candidate] (9.283 s) : 0, 9282932
gantt
title insecure-bank - break down per module: candidate=1.59.0-SNAPSHOT~078e8d1842, baseline=1.59.0-SNAPSHOT~372ceb0ed6
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.208 ms) : 0, 1208
crashtracking [candidate] (1.185 ms) : 0, 1185
BytebuddyAgent [baseline] (656.541 ms) : 0, 656541
BytebuddyAgent [candidate] (652.476 ms) : 0, 652476
GlobalTracer [baseline] (284.206 ms) : 0, 284206
GlobalTracer [candidate] (282.958 ms) : 0, 282958
AppSec [baseline] (33.121 ms) : 0, 33121
AppSec [candidate] (32.674 ms) : 0, 32674
Debugger [baseline] (68.003 ms) : 0, 68003
Debugger [candidate] (67.409 ms) : 0, 67409
Remote Config [baseline] (606.591 µs) : 0, 607
Remote Config [candidate] (613.051 µs) : 0, 613
Telemetry [baseline] (8.994 ms) : 0, 8994
Telemetry [candidate] (8.937 ms) : 0, 8937
Flare Poller [baseline] (3.795 ms) : 0, 3795
Flare Poller [candidate] (3.713 ms) : 0, 3713
section iast
crashtracking [baseline] (1.188 ms) : 0, 1188
crashtracking [candidate] (1.19 ms) : 0, 1190
BytebuddyAgent [baseline] (792.385 ms) : 0, 792385
BytebuddyAgent [candidate] (792.885 ms) : 0, 792885
GlobalTracer [baseline] (256.465 ms) : 0, 256465
GlobalTracer [candidate] (256.598 ms) : 0, 256598
AppSec [baseline] (34.272 ms) : 0, 34272
AppSec [candidate] (35.469 ms) : 0, 35469
Debugger [baseline] (64.956 ms) : 0, 64956
Debugger [candidate] (64.29 ms) : 0, 64290
Remote Config [baseline] (579.625 µs) : 0, 580
Remote Config [candidate] (602.469 µs) : 0, 602
Telemetry [baseline] (8.443 ms) : 0, 8443
Telemetry [candidate] (8.576 ms) : 0, 8576
Flare Poller [baseline] (3.561 ms) : 0, 3561
Flare Poller [candidate] (3.599 ms) : 0, 3599
IAST [baseline] (27.07 ms) : 0, 27070
IAST [candidate] (27.247 ms) : 0, 27247
Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.59.0-SNAPSHOT~078e8d1842, baseline=1.59.0-SNAPSHOT~372ceb0ed6
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.085 s) : 0, 1085120
Total [baseline] (10.822 s) : 0, 10821510
Agent [candidate] (1.086 s) : 0, 1086433
Total [candidate] (10.845 s) : 0, 10844873
section appsec
Agent [baseline] (1.27 s) : 0, 1269812
Total [baseline] (10.934 s) : 0, 10933860
Agent [candidate] (1.275 s) : 0, 1274843
Total [candidate] (10.983 s) : 0, 10983082
section iast
Agent [baseline] (1.227 s) : 0, 1226754
Total [baseline] (11.198 s) : 0, 11198245
Agent [candidate] (1.233 s) : 0, 1232966
Total [candidate] (11.128 s) : 0, 11127603
section profiling
Agent [baseline] (1.215 s) : 0, 1214729
Total [baseline] (11.001 s) : 0, 11001195
Agent [candidate] (1.211 s) : 0, 1211382
Total [candidate] (10.904 s) : 0, 10904268
gantt
title petclinic - break down per module: candidate=1.59.0-SNAPSHOT~078e8d1842, baseline=1.59.0-SNAPSHOT~372ceb0ed6
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.179 ms) : 0, 1179
crashtracking [candidate] (1.187 ms) : 0, 1187
BytebuddyAgent [baseline] (651.173 ms) : 0, 651173
BytebuddyAgent [candidate] (652.472 ms) : 0, 652472
GlobalTracer [baseline] (282.454 ms) : 0, 282454
GlobalTracer [candidate] (282.685 ms) : 0, 282685
AppSec [baseline] (32.637 ms) : 0, 32637
AppSec [candidate] (32.805 ms) : 0, 32805
Debugger [baseline] (68.69 ms) : 0, 68690
Debugger [candidate] (68.364 ms) : 0, 68364
Remote Config [baseline] (624.906 µs) : 0, 625
Remote Config [candidate] (628.515 µs) : 0, 629
Telemetry [baseline] (9.048 ms) : 0, 9048
Telemetry [candidate] (8.992 ms) : 0, 8992
Flare Poller [baseline] (3.835 ms) : 0, 3835
Flare Poller [candidate] (3.748 ms) : 0, 3748
section appsec
crashtracking [baseline] (1.18 ms) : 0, 1180
crashtracking [candidate] (1.188 ms) : 0, 1188
BytebuddyAgent [baseline] (693.541 ms) : 0, 693541
BytebuddyAgent [candidate] (696.498 ms) : 0, 696498
GlobalTracer [baseline] (259.234 ms) : 0, 259234
GlobalTracer [candidate] (260.127 ms) : 0, 260127
AppSec [baseline] (174.795 ms) : 0, 174795
AppSec [candidate] (175.558 ms) : 0, 175558
Debugger [baseline] (66.865 ms) : 0, 66865
Debugger [candidate] (66.971 ms) : 0, 66971
Remote Config [baseline] (746.446 µs) : 0, 746
Remote Config [candidate] (777.13 µs) : 0, 777
Telemetry [baseline] (9.498 ms) : 0, 9498
Telemetry [candidate] (9.436 ms) : 0, 9436
Flare Poller [baseline] (3.753 ms) : 0, 3753
Flare Poller [candidate] (3.788 ms) : 0, 3788
IAST [baseline] (24.634 ms) : 0, 24634
IAST [candidate] (24.893 ms) : 0, 24893
section iast
crashtracking [baseline] (1.187 ms) : 0, 1187
crashtracking [candidate] (1.192 ms) : 0, 1192
BytebuddyAgent [baseline] (792.67 ms) : 0, 792670
BytebuddyAgent [candidate] (797.785 ms) : 0, 797785
GlobalTracer [baseline] (256.682 ms) : 0, 256682
GlobalTracer [candidate] (257.938 ms) : 0, 257938
AppSec [baseline] (34.526 ms) : 0, 34526
AppSec [candidate] (34.023 ms) : 0, 34023
Debugger [baseline] (66.115 ms) : 0, 66115
Debugger [candidate] (66.521 ms) : 0, 66521
Remote Config [baseline] (577.672 µs) : 0, 578
Remote Config [candidate] (569.555 µs) : 0, 570
Telemetry [baseline] (8.603 ms) : 0, 8603
Telemetry [candidate] (8.487 ms) : 0, 8487
Flare Poller [baseline] (3.578 ms) : 0, 3578
Flare Poller [candidate] (3.484 ms) : 0, 3484
IAST [baseline] (27.343 ms) : 0, 27343
IAST [candidate] (27.34 ms) : 0, 27340
section profiling
ProfilingAgent [baseline] (98.431 ms) : 0, 98431
ProfilingAgent [candidate] (96.779 ms) : 0, 96779
crashtracking [baseline] (1.233 ms) : 0, 1233
crashtracking [candidate] (1.222 ms) : 0, 1222
BytebuddyAgent [baseline] (708.781 ms) : 0, 708781
BytebuddyAgent [candidate] (707.633 ms) : 0, 707633
GlobalTracer [baseline] (221.557 ms) : 0, 221557
GlobalTracer [candidate] (221.741 ms) : 0, 221741
AppSec [baseline] (32.433 ms) : 0, 32433
AppSec [candidate] (32.287 ms) : 0, 32287
Debugger [baseline] (68.765 ms) : 0, 68765
Debugger [candidate] (68.346 ms) : 0, 68346
Remote Config [baseline] (649.149 µs) : 0, 649
Remote Config [candidate] (640.719 µs) : 0, 641
Telemetry [baseline] (8.886 ms) : 0, 8886
Telemetry [candidate] (8.847 ms) : 0, 8847
Flare Poller [baseline] (3.687 ms) : 0, 3687
Flare Poller [candidate] (3.701 ms) : 0, 3701
Profiling [baseline] (99.018 ms) : 0, 99018
Profiling [candidate] (97.346 ms) : 0, 97346
LoadParameters
See matching parameters
SummaryFound 2 performance improvements and 1 performance regressions! Performance is the same for 17 metrics, 16 unstable metrics.
Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.59.0-SNAPSHOT~078e8d1842, baseline=1.59.0-SNAPSHOT~372ceb0ed6
dateFormat X
axisFormat %s
section baseline
no_agent (17.292 ms) : 17114, 17469
. : milestone, 17292,
appsec (19.532 ms) : 19335, 19729
. : milestone, 19532,
code_origins (17.625 ms) : 17449, 17802
. : milestone, 17625,
iast (17.732 ms) : 17560, 17905
. : milestone, 17732,
profiling (19.114 ms) : 18922, 19307
. : milestone, 19114,
tracing (17.658 ms) : 17483, 17833
. : milestone, 17658,
section candidate
no_agent (19.322 ms) : 19124, 19521
. : milestone, 19322,
appsec (18.658 ms) : 18471, 18845
. : milestone, 18658,
code_origins (17.834 ms) : 17656, 18012
. : milestone, 17834,
iast (17.71 ms) : 17531, 17889
. : milestone, 17710,
profiling (18.54 ms) : 18354, 18726
. : milestone, 18540,
tracing (17.892 ms) : 17715, 18068
. : milestone, 17892,
Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.59.0-SNAPSHOT~078e8d1842, baseline=1.59.0-SNAPSHOT~372ceb0ed6
dateFormat X
axisFormat %s
section baseline
no_agent (1.195 ms) : 1183, 1206
. : milestone, 1195,
iast (3.114 ms) : 3076, 3152
. : milestone, 3114,
iast_FULL (5.906 ms) : 5847, 5966
. : milestone, 5906,
iast_GLOBAL (3.587 ms) : 3536, 3638
. : milestone, 3587,
profiling (1.937 ms) : 1919, 1955
. : milestone, 1937,
tracing (1.891 ms) : 1875, 1907
. : milestone, 1891,
section candidate
no_agent (1.187 ms) : 1175, 1198
. : milestone, 1187,
iast (3.081 ms) : 3042, 3119
. : milestone, 3081,
iast_FULL (5.747 ms) : 5689, 5804
. : milestone, 5747,
iast_GLOBAL (3.605 ms) : 3553, 3657
. : milestone, 3605,
profiling (1.997 ms) : 1979, 2015
. : milestone, 1997,
tracing (1.787 ms) : 1772, 1802
. : milestone, 1787,
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 10 metrics, 2 unstable metrics. Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.59.0-SNAPSHOT~078e8d1842, baseline=1.59.0-SNAPSHOT~372ceb0ed6
dateFormat X
axisFormat %s
section baseline
no_agent (1.481 ms) : 1470, 1493
. : milestone, 1481,
appsec (3.738 ms) : 3517, 3959
. : milestone, 3738,
iast (2.225 ms) : 2160, 2290
. : milestone, 2225,
iast_GLOBAL (2.281 ms) : 2215, 2346
. : milestone, 2281,
profiling (2.468 ms) : 2251, 2685
. : milestone, 2468,
tracing (2.054 ms) : 2002, 2105
. : milestone, 2054,
section candidate
no_agent (1.479 ms) : 1467, 1490
. : milestone, 1479,
appsec (3.751 ms) : 3525, 3977
. : milestone, 3751,
iast (2.226 ms) : 2161, 2292
. : milestone, 2226,
iast_GLOBAL (2.269 ms) : 2204, 2335
. : milestone, 2269,
profiling (2.088 ms) : 2034, 2143
. : milestone, 2088,
tracing (2.051 ms) : 2000, 2102
. : milestone, 2051,
Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.59.0-SNAPSHOT~078e8d1842, baseline=1.59.0-SNAPSHOT~372ceb0ed6
dateFormat X
axisFormat %s
section baseline
no_agent (15.259 s) : 15259000, 15259000
. : milestone, 15259000,
appsec (14.807 s) : 14807000, 14807000
. : milestone, 14807000,
iast (18.369 s) : 18369000, 18369000
. : milestone, 18369000,
iast_GLOBAL (17.829 s) : 17829000, 17829000
. : milestone, 17829000,
profiling (15.363 s) : 15363000, 15363000
. : milestone, 15363000,
tracing (14.656 s) : 14656000, 14656000
. : milestone, 14656000,
section candidate
no_agent (15.677 s) : 15677000, 15677000
. : milestone, 15677000,
appsec (14.493 s) : 14493000, 14493000
. : milestone, 14493000,
iast (18.066 s) : 18066000, 18066000
. : milestone, 18066000,
iast_GLOBAL (18.228 s) : 18228000, 18228000
. : milestone, 18228000,
profiling (14.83 s) : 14830000, 14830000
. : milestone, 14830000,
tracing (14.675 s) : 14675000, 14675000
. : milestone, 14675000,
|
| // Pattern to match Kafka Connect schema-derived suffixes like _messagebody_0, _text_0, _bytes_0 | ||
| // These suffixes are added by Kafka Connect converters when handling union/optional fields | ||
| private static final java.util.regex.Pattern KAFKA_CONNECT_SCHEMA_SUFFIX_PATTERN = | ||
| java.util.regex.Pattern.compile("_(?:messagebody|text|bytes|map|value)_\\d+$", java.util.regex.Pattern.CASE_INSENSITIVE); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🎯 suggestion: Pattern usage is not recommended for performance reason.
It seems the pattern you're looking for would be easy to find as the last characters would be a number.
I would implement the clean up using String.lastIndexOf() and String.substring() instead of Pattern, Matcher and String.replaceFirst() which also use a pattern internally.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sounds good, updated it to no longer use a regex pattern.
…mance Address review feedback to avoid Pattern/Matcher usage for performance reasons. Use lastIndexOf() and regionMatches() instead of regex to strip Kafka Connect schema-derived suffixes from queue names. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
|
Hi! 👋 Thanks for your pull request! 🎉 To help us review it, please make sure to:
If you need help, please check our contributing guidelines. |
PerfectSlayer
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looking good, thanks for the follow up changes
...entation/jms/javax-jms-1.1/src/main/java/datadog/trace/instrumentation/jms/JMSDecorator.java
Outdated
Show resolved
Hide resolved
Group len==5 cases together (bytes, value) before len==3 case (map). Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Summary
When using Kafka Connect with IBM MQ connectors, DSM was reporting incorrect queue names with schema-derived suffixes like
_messagebody_0. For example:trainmgt.dispatch.trnsheet.p30.v1.pub_messagebody_0trainmgt.dispatch.trnsheet.p30.v1.pubHow IBM MQ and Kafka Connect are linked
Kafka Connect IBM MQ connectors (from Confluent or IBM) bridge IBM MQ and Kafka:
The connector internally uses JMS to communicate with IBM MQ. When dd-trace-java is attached to the Kafka Connect worker JVM, it instruments these JMS calls and creates DSM checkpoints using
Queue.getQueueName().Hypothesis
When the connector creates JMS Queue objects, somehow the queue name returned by
getQueueName()includes schema-derived suffixes. This could happen if:_0suffixes to union/optional fields likemessageBody, and this leaks into the destination nameWhy pure Java apps work
Pure Java apps create queues directly:
With Kafka Connect, the connector creates the Queue internally, and something in that process adds the suffix.
Changes
Added sanitization in
JMSDecorator.getDestinationName()to strip these suffixes:_messagebody_N_text_N_bytes_N_map_N_value_NUses a string-based approach with
lastIndexOf()andregionMatches()for performance (avoids Pattern/Matcher overhead).Test plan
JMSDecoratorTest.groovycovering:References
🤖 Generated with Claude Code