Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 10 additions & 5 deletions config/_default/menus/main.en.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -7051,31 +7051,36 @@ menu:
parent: observability_pipelines_destinations
identifier: observability_pipelines_opensearch
weight: 420
- name: OpenTelemetry
url: observability_pipelines/destinations/opentelemetry
parent: observability_pipelines_destinations
identifier: observability_pipelines_destinations_opentelemetry
weight: 421
- name: SentinelOne
url: observability_pipelines/destinations/sentinelone
parent: observability_pipelines_destinations
identifier: observability_pipelines_sentinelone
weight: 421
weight: 422
- name: Socket
url: observability_pipelines/destinations/socket
parent: observability_pipelines_destinations
identifier: observability_pipelines_socket
weight: 422
weight: 423
- name: Splunk HEC
url: observability_pipelines/destinations/splunk_hec/logs/
parent: observability_pipelines_destinations
identifier: observability_pipelines_splunk_hec
weight: 423
weight: 424
- name: Sumo Logic Hosted Collector
url: observability_pipelines/destinations/sumo_logic_hosted_collector
parent: observability_pipelines_destinations
identifier: observability_pipelines_sumo_logic_hosted_collector
weight: 424
weight: 425
- name: Syslog
url: observability_pipelines/destinations/syslog
parent: observability_pipelines_destinations
identifier: observability_pipelines_syslog
weight: 425
weight: 426
- name: Packs
url: observability_pipelines/packs/
parent: observability_pipelines
Expand Down
2 changes: 2 additions & 0 deletions content/en/observability_pipelines/destinations/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,12 +77,14 @@ These are the available destinations:
- [Datadog Metrics][1]
- [Elasticsearch][2]
- [HTTP/S Client][3]
- [OpenTelemetry][5]
- [Splunk HEC][4]

[1]: /observability_pipelines/destinations/datadog_metrics/
[2]: /observability_pipelines/destinations/elasticsearch/
[3]: /observability_pipelines/destinations/http_client/
[4]: /observability_pipelines/destinations/splunk_hec/metrics
[5]: /observability_pipelines/destinations/opentelemetry/

{{% /tab %}}
{{< /tabs >}}
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
---
title: OpenTelemetry Destination
disable_toc: false
products:
- name: Metrics
icon: metrics
url: /observability_pipelines/configuration/?tab=metrics#pipeline-types
---

{{< product-availability >}}

## Overview

Use Observability Pipelines' OpenTelemetry destination to send metrics over HTTP/S to an OpenTelemetry (OTEL) Collector or another OpenTelemetry Protocol (OTLP)-compatible endpoint.

## Set up destination

<div class="alert alert-danger">For Secrets Management: Only enter the identifier for the HTTP/S Client URI and, if applicable, the TLS key pass. Do <b>not</b> enter the actual values.</div>

Configure the OpenTelemetry destination when you [set up a pipeline][3]. You can set up a pipeline in the [UI][1], using the [API][4], or with [Terraform][5]. The steps in this section are configured in the UI.

After you select the OpenTelemetry destination in the pipeline UI, enter the identifier for your HTTP/S Client URI. If you leave it blank, the [default](#secret-defaults) is used.

**Notes**:
- The Worker can only send counter, gauge, and histogram metrics to OpenTelemetry. OpenTelemetry does not support other metrics types so the Worker drops them. See [Filter out unsupported metrics](#filter-out-unsupported-metrics) for more information.
- If you are sending your metrics to a Prometheus OTLP receiver, set your OTLP receiver to allow out-of-order samples. Otherwise, the Prometheus OTLP receiver rejects the out of order sample and the Worker logs a bad request error and drops the batch of metrics.
- If you enter secret identifiers and then choose to use environment variables, the environment variable is the identifier entered and prepended with `DD_OP_`. For example, if you entered `PASSWORD_1` for a password identifier, the environment variable for that password is `DD_OP_PASSWORD_1`.

### Optional settings

#### Enable TLS

{{% observability_pipelines/tls_settings %}}

#### Buffering

{{% observability_pipelines/destination_buffer %}}

## Filter out unsupported metrics

The Worker can only send counter, gauge, and histogram metrics to OpenTelemetry. The following Datadog metrics are not supported because they cannot be converted to OTLP format:

- StatsD-type metrics
- Distribution metrics
- Sketch metrics

If one of these metrics is in a batch to be encoded and sent to OpenTelemetry, the Worker drops the unsupported metric, logs an error, and updates the `component_error_total` metric. Datadog recommends using a [filter processor][9] to filter out unsupported metric types.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to confirm with eng that the unsupported metric gets dropped and not the whole batch.


## Secret defaults

{{% observability_pipelines/set_secrets_intro %}}

{{< tabs >}}
{{% tab "Secrets Management" %}}

- HTTP/S Client URI endpoint identifier
- References the HTTP/S URI endpoint to which the Worker sends OpenTelemetry data. For example: `http://localhost:4319/v1/metrics`.
- The default identifier is `DESTINATION_OTEL_HTTP_CLIENT_URI`.
- HTTP/S Client TLS passphrase identifier (when TLS is enabled):
- The default identifier is `DESTINATION_OTEL_HTTP_CLIENT_KEY_PASS`.

{{% /tab %}}

{{% tab "Environment Variables" %}}

{{% observability_pipelines/configure_existing_pipelines/destination_env_vars/opentelemetry %}}

{{% /tab %}}
{{< /tabs >}}

## Metrics

For [component metrics][6] and [destination buffer metrics][7] emitted by all destinations, see the [Pipelines Usage Metrics][8] documentation. To filter or group by OpenTelemetry destination metrics, use the tag `component_type:opentelemetry`.

## How the destination works

### Event batching

A batch of events is flushed when one of these conditions occurs. See [event batching][2] for more information.

| Maximum Events | Maximum Size (MB) | Timeout (seconds) |
|----------------|-------------------|---------------------|
| TKTK | TKTK | TKTK |

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Waiting for eng on this


[1]: https://app.datadoghq.com/observability-pipelines
[2]: /observability_pipelines/destinations/#event-batching
[3]: /observability_pipelines/configuration/set_up_pipelines/
[4]: /api/latest/observability-pipelines/
[5]: https://registry.terraform.io/providers/datadog/datadog/latest/docs/resources/observability_pipeline
[6]: /observability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics/#component-metrics
[7]: /observability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics/#destination-buffer-metrics
[8]: /observability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics/
[9]: /observability_pipelines/processors/filter/
Original file line number Diff line number Diff line change
Expand Up @@ -125,6 +125,9 @@ Some Observability Pipelines components require setting up environment variables
### OpenSearch
{{% observability_pipelines/configure_existing_pipelines/destination_env_vars/opensearch %}}

### OpenTelemetry
{{% observability_pipelines/configure_existing_pipelines/destination_env_vars/opentelemetry %}}

### SentinelOne
{{% observability_pipelines/configure_existing_pipelines/destination_env_vars/sentinelone %}}

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
- HTTP/S client URI endpoint:
- The HTTP/S URI endpoint to which the Worker sends OpenTelemetry data. For example: `http://localhost:4319/v1/metrics`.
- The default environment variable is `DD_OP_DESTINATION_OTEL_HTTP_CLIENT_URI`.
- HTTP/S Client TLS passphrase (when enabled):
- The default environment variable is `DD_OP_DESTINATION_OTEL_HTTP_CLIENT_KEY_PASS`.
Loading