Automatic trace propagation stops working after ZIO upgrade to 2.1.20

[paul-danilin-moia]

Hi all,
we are using

"dev.zio" %% "zio-opentelemetry" % "3.1.7"
"io.opentelemetry.javaagent" % "opentelemetry-javaagent" % "2.18.1"

and noticed that updating

"dev.zio" %% "zio" % <version>

from 2.1.19 to 2.1.20 breaks propagation of the tracing context. Roughly speaking this is what we encounter:

val tracingLayer = (OpenTelemetry.global ++ OpenTelemetry.contextJVM) >>> ZioOpenTelemetry.tracing(instrumentationScopeName = "some-name")
val tracing: Tracing = ??? // obtained through tracingLayer

// when handling an HTTP request via http4s
tracing.root("span-name") {
  tracing.getCurrentSpanContextUnsafe.debug("before") *>
  http4sRequest.decodeStrict[SomeType] { parsedRequest =>
    tracing.getCurrentSpanContextUnsafe.debug("after") *>
    ???
  }
}

The generated span context is witnessed in before but is lost in after (the one logged at after is always 000000).
The span is propagated correctly if we don't update the dependencies to 2.1.20 or update everything to the latest and add dependencyOverrides += "dev.zio" %% "zio" % "2.1.19"

I tried reproducing the issue independently but I haven't succeeded so far. And I understand it is only remotely helpful without a reproducer. Do you probably have an idea, what can be involved here?
Maybe someone else has a similar problem and would be more successful with a reproducer?

[grouzen]

Hello! Thanks for reporting!
I must say that I consider the ZIO instrumentation in opentelemetry-javaagent broken. This is the reason I'm developing the next version 4.0.0 of the library. You can check out the latest RC release: https://github.com/zio/zio-telemetry/releases/tag/v4.0.0-RC7

The examples in v4.0.0-rc branch are updated so you can have an idea what's changed.
I'm very curious if switching to the 4.0.0-RC version fixes your issue. Do you think it is possible to switch your code to the newer version to see how it goes?

I tried reproducing the issue independently but I haven't succeeded so far. And I understand it is only remotely helpful without a reproducer. Do you probably have an idea, what can be involved here?

I think the first thing to try is adding a test to reproduce it. I believe it should be doable, just need to figure out how to make openetelemetry java-agent work with our tests.

Maybe someone else has a similar problem and would be more successful with a reproducer?

Unfortunately, I'm not aware of it. You can try asking in the Discord channel: https://discord.com/channels/629491597070827530/639825316021272602

[grouzen]

@paul-danilin-moia
I just realized, what if you remove the http4s.decodeStrict call from your code:

tracing.root("span-name") {
  tracing.getCurrentSpanContextUnsafe.debug("before") *> tracing.getCurrentSpanContextUnsafe.debug("after")
}

[paul-danilin-moia]

@paul-danilin-moia I just realized, what if you remove the http4s.decodeStrict call from your code:

tracing.root("span-name") {
  tracing.getCurrentSpanContextUnsafe.debug("before") *> tracing.getCurrentSpanContextUnsafe.debug("after")
}

Removing decodeStrict fixes the context propagation. What bugs me though: 1) why the behaviour is broken by a patch update in zio runtime. 2) why the issue is not reproduced on a simpler "sandboxed" version

[paul-danilin-moia]

Hello! Thanks for reporting! I must say that I consider the ZIO instrumentation in opentelemetry-javaagent broken. This is the reason I'm developing the next version 4.0.0 of the library. You can check out the latest RC release: https://github.com/zio/zio-telemetry/releases/tag/v4.0.0-RC7

The examples in v4.0.0-rc branch are updated so you can have an idea what's changed. I'm very curious if switching to the 4.0.0-RC version fixes your issue. Do you think it is possible to switch your code to the newer version to see how it goes?

It definitely is possible. I will try the 4.0.0 RC version on our project within a couple days and get back to you.

I tried reproducing the issue independently but I haven't succeeded so far. And I understand it is only remotely helpful without a reproducer. Do you probably have an idea, what can be involved here?

I think the first thing to try is adding a test to reproduce it. I believe it should be doable, just need to figure out how to make openetelemetry java-agent work with our tests.

Yep, the issue is that I was able to localize it but not to create a reproducible outside of our closed source project 😢

[guizmaii]

Could the issue be with the ZIO/Cats integration? I remember seeing such an issue with context propagation between Cats and ZIO in the past already (but wasn't able to find any trace sadly)

[grouzen]

Could the issue be with the ZIO/Cats integration? I remember seeing such an issue with context propagation between Cats and ZIO in the past already (but wasn't able to find any trace sadly)

Yes, it definitely could be the case.
The issue may also be caused by the interop between the ZIO runtime and the OTEL javaagent ZIO module

[grouzen]

@paul-danilin-moia

It definitely is possible. I will try the 4.0.0 RC version on our project within a couple days and get back to you.

I just realised I've not explicitly added to the scala doc of the OpenTelemetry.autoinstrumented method the note about disabling zio instrumentation in OTEL javaagent. It can be achieved by adding the following sbt command line argument -J-Dotel.instrumentation.zio.enabled=false as mentioned in https://github.com/zio/zio-telemetry/blob/v4.0.0-rc/docs/opentelemetry-instrumentation-example.md

[grouzen]

Found the link in my notes about:

I must say that I consider the ZIO instrumentation in opentelemetry-javaagent broken.

open-telemetry/opentelemetry-java#5303

[grouzen]

To summarise.

The potential points of failure:

• ZIO instrumentation in OTEL javaagent
• cats-effect interop

It is hard to debug cats interop without being able to reproduce the issue.
Given this, the first option to try is to stop dealing with ZIO instrumentation. For this, you need to switch to version 4.0.0-RC of the library.

[paul-danilin-moia]

So, I tried switching to 4.0.0-RC7. The code now looks something like this:

val tracingLayer = OpenTelemetry.global >+>OpenTelemetry.tracer(instrumentationScopeName = "scope")
val tracer: Tracer = ??? // obtained through tracingLayer
val opentelemetry: OpenTelemetry = ??? // obtained through tracingLayer

// when handling an HTTP request via http4s
tracing.root("span-name") {
  opentelemetry.unsafe.getCurrentContext.debug("before") *>
  http4sRequest.decodeStrict[SomeType] { parsedRequest =>
    opentelemetry.unsafe.getCurrentContext.debug("after") *>
    ???
  } @@ opentelemetry.aspects.autoinstrumented
}

with sbt setup javaOptions += "-Dotel.instrumentation.zio.enabled=false".
The issue persists with output roughly:

before: AgentContextWrapper{agentContext={opentelemetry-trace-span-key=SdkSpan{traceId=ad5ea614022112940a14f258e4333d06, spanId=9e429f515e632780, <...>}
after: AgentContextWrapper{agentContext={}, applicationContext={}}

[grouzen]

@paul-danilin-moia Which versions of http4s and cats-interop libraries are you on?

[paul-danilin-moia]

"org.http4s" %% "http4s-core" % "0.23.30"
"dev.zio" %% "zio-interop-cats" % "23.1.0.5"

[masonedmison]

Hello! I suspect we at my company are running up against this issue. @grouzen Trying to wrap my head around what's known based off of what's been discussed above. Since, here, the issues persists with "-Dotel.instrumentation.zio.enabled=false", with the version 4.0.0, do we suspect that this issue doesn't have anything to do with the opentelemetry-java-instrumentation?