AWS DynamoDB traces split with other services after instrumented for Lambda NodeJS application by self-building a layer

[WinfredWW]

What version of OpenTelemetry are you using?

"@opentelemetry/instrumentation-aws-lambda": "^0.61.0",
"@opentelemetry/instrumentation-aws-sdk": "^0.64.0",
"@opentelemetry/instrumentation-graphql": "^0.56.0",
"@opentelemetry/instrumentation-mysql": "^0.54.0",
"@opentelemetry/instrumentation-mysql2": "^0.55.0",
"@opentelemetry/instrumentation-redis": "^0.57.1",

What version of Node are you using?

NodeJS: 22.20.0

What did you do?

We used this to build a Lambda Layer for running NodeJS applications, add this layer to Lambda, and we sent the OpenTelemetry traces to OpenTelemetry Collector. https://github.com/open-telemetry/opentelemetry-lambda/blob/main/nodejs/README.md

The Lambda application has interactions with AWS Redis, RDS MySQL and DynamoDB, but the traces split between application/Redis/MySQL and DynamoDB (aka. different traceIDs).

Here are the environment variables we configured for Lambda:

AWS_LAMBDA_EXEC_WRAPPER
/opt/otel-handler
OTEL_EXPORTER_OTLP_ENDPOINT
http:///otel
OTEL_EXPORTER_OTLP_TRACES_PROTOCOL
http/protobuf
OTEL_LOG_LEVEL
debug
OTEL_NODE_ENABLED_INSTRUMENTATIONS
aws-lambda,aws-sdk,http,pg,mysql,mysql2,redis,ioredis,graphql,nestjs-core,connect,http,dns,express
OTEL_TRACES_SAMPLER
always_on

What did you expect to see?

The same trace from application to all other AWS services

What did you see instead?

In the OpemTelemetry Collector log file as attached, I can see there are two trace IDs, so it seems like the trace context is not successfully propagated during the Lambda-to-Lambda invocation process. Except the method to explicitly pass the context in the code (since it is not feasible to change every Lambda code), is there any other way to propagate the traces automatically (when building the layer by myself as ADOT is not available here)

Additional context

Log file attached

collectorlog.txt

[trentm]

so it seems like the trace context is not successfully propagated during the Lambda-to-Lambda invocation process

@WinfredWW Can you describe your Lambda architecture? What is in one Lambda that is calling another Lambda? What is the trigger for the second Lambda? Is it an HTTP trigger call? Perhaps a trigger on a DynamoDB addition (I'm not even sure if Lambda supports DynamoDB triggers)? Something else?

[pichlermarc]

@WinfredWW the question around the architecture is relevant as we need to know if the path is actually instrumented to extract/inject context.

[WinfredWW]

Hi @trentm @pichlermarc, thank you for your reply, and sorry for my late response due to vacations and holidays here. As for the Lambda architecture, I will need to confirm with my customers and then get back here.