OpenTelemetry.zioMetrics metrics are dropped by OpenTelemetry Collector

[kdelija]

When enabling OpenTelemetry.zioMetrics some of the metrics attributes are sent with empty values.
In this example, you can see that scopeMetrics / metrics / dataPoints/attributes have a key called bucket with an empty value.

  "resource": {
    "attributes": [
      {
        "key": "service.name",
        "value": {
          "stringValue": "test-service"
        }
      }
    ]
  },
  "scopeMetrics": [
    {
      "scope": {
        "name": "com.test",
        "attributes": []
      },
      "metrics": [
        {
          "name": "zio_fiber_fork_locations",
          "sum": {
            "dataPoints": [
              {
                "startTimeUnixNano": "1741701483845719000",
                "timeUnixNano": "1741702203864049000",
                "asInt": "3",
                "exemplars": [],
                "attributes": [
                  {
                    "key": "bucket",
                    "value": {}
                  }
                ]
              },
              {
                "startTimeUnixNano": "1741701483845719000",
                "timeUnixNano": "1741702203864049000",
                "asInt": "36",
                "exemplars": [],
                "attributes": [
                  {
                    "key": "bucket",
                    "value": {
                      "stringValue": "com.test.HttpServer.start(HttpServer.scala:41)"
                    }
                  }
                ]
              }
            ],
            "aggregationTemporality": 2,
            "isMonotonic": true
          }
        }
      ]
    }
  ]
}

This metric is then dropped by OpenTelemetry Collector with this log message:

ts=2025-03-11T12:59:43.467663562Z caller=handler.go:1041 level=debug component=receive component=receive-handler msg="failed to handle request" err="add 1 series: label set contains a label with empty name or value"
ts=2025-03-11T12:59:43.468372376Z caller=writer_errors.go:45 level=debug component=receive component=receive-writer tenant=default-tenant msg="Labels with empty name in the label set" lset="labels:<name:"name" value:"zio_fiber_fork_locations_total" > labels:<name:"bucket" > labels:<name:"job" value:"test-service" > labels:<name:"namespace" value:"test" > labels:<name:"pod" value:"test-5f99f4b77c-vjh79" > "
ts=2025-03-11T12:59:43.468472726Z caller=writer_errors.go:127 level=warn component=receive component=receive-writer tenant=default-tenant msg="Error on series with empty label name or value" numDropped=1
ts=2025-03-11T12:59:43.468636147Z caller=handler.go:1041 level=debug component=receive component=receive-handler msg="failed to handle request" err="add 1 series: label set contains a label with empty name or value"

Project dependencies:

1. zio 2.1.14
2. zio-opentelemetry 3.1.1
3. opentelemetry-sdk 1.47.0
4. opentelemetry-exporter-otlp 1.47.0
5. opentelemetry-exporter-logging-otlp 1.47.0
6. opentelemetry-semconv 1 .30.0

[grouzen]

Hello! Thanks for reporting!

[grouzen]

I can confirm it is reproducible on my machine with the same problematic metric zio_fiber_fork_locations.
It likely requires digging into https://github.com/zio/zio/tree/series/2.x/core/shared/src/main/scala/zio/metrics

[jivanic-demystdata]

Is this bug present in v4? 🤔

[grouzen]

Is this bug present in v4? 🤔

Hey! I somehow missed this comment, sorry.
Yes, I think it is still there and needs to be addressed. The reason is that it is caused by the ZIO metrics, not by zio-telemetry.