-
Notifications
You must be signed in to change notification settings - Fork 1.3k
[DOCS-15273] Add OpenTelemetry metrics destination #38656
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
maycmlee
wants to merge
5
commits into
master
Choose a base branch
from
may/otel-metrics-destination
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
93 changes: 93 additions & 0 deletions
93
content/en/observability_pipelines/destinations/opentelemetry.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,93 @@ | ||
| --- | ||
| title: OpenTelemetry Destination | ||
| disable_toc: false | ||
| products: | ||
| - name: Metrics | ||
| icon: metrics | ||
| url: /observability_pipelines/configuration/?tab=metrics#pipeline-types | ||
| --- | ||
|
|
||
| {{< product-availability >}} | ||
|
|
||
| ## Overview | ||
|
|
||
| Use Observability Pipelines' OpenTelemetry destination to send metrics over HTTP/S to an OpenTelemetry (OTEL) Collector or another OpenTelemetry Protocol (OTLP)-compatible endpoint. | ||
|
|
||
| ## Set up destination | ||
|
|
||
| <div class="alert alert-danger">For Secrets Management: Only enter the identifier for the HTTP/S Client URI and, if applicable, the TLS key pass. Do <b>not</b> enter the actual values.</div> | ||
|
|
||
| Configure the OpenTelemetry destination when you [set up a pipeline][3]. You can set up a pipeline in the [UI][1], using the [API][4], or with [Terraform][5]. The steps in this section are configured in the UI. | ||
|
|
||
| After you select the OpenTelemetry destination in the pipeline UI, enter the identifier for your HTTP/S Client URI. If you leave it blank, the [default](#secret-defaults) is used. | ||
|
|
||
| **Notes**: | ||
| - The Worker can only send counter, gauge, and histogram metrics to OpenTelemetry. OpenTelemetry does not support other metrics types so the Worker drops them. See [Filter out unsupported metrics](#filter-out-unsupported-metrics) for more information. | ||
| - If you are sending your metrics to a Prometheus OTLP receiver, set your OTLP receiver to allow out-of-order samples. Otherwise, the Prometheus OTLP receiver rejects the out of order sample and the Worker logs a bad request error and drops the batch of metrics. | ||
| - If you enter secret identifiers and then choose to use environment variables, the environment variable is the identifier entered and prepended with `DD_OP_`. For example, if you entered `PASSWORD_1` for a password identifier, the environment variable for that password is `DD_OP_PASSWORD_1`. | ||
|
|
||
| ### Optional settings | ||
|
|
||
| #### Enable TLS | ||
|
|
||
| {{% observability_pipelines/tls_settings %}} | ||
|
|
||
| #### Buffering | ||
|
|
||
| {{% observability_pipelines/destination_buffer %}} | ||
|
|
||
| ## Filter out unsupported metrics | ||
|
|
||
| The Worker can only send counter, gauge, and histogram metrics to OpenTelemetry. The following Datadog metrics are not supported because they cannot be converted to OTLP format: | ||
|
|
||
| - StatsD-type metrics | ||
| - Distribution metrics | ||
| - Sketch metrics | ||
|
|
||
| If one of these metrics is in a batch to be encoded and sent to OpenTelemetry, the Worker drops the unsupported metric, logs an error, and updates the `component_error_total` metric. Datadog recommends using a [filter processor][9] to filter out unsupported metric types. | ||
|
|
||
| ## Secret defaults | ||
|
|
||
| {{% observability_pipelines/set_secrets_intro %}} | ||
|
|
||
| {{< tabs >}} | ||
| {{% tab "Secrets Management" %}} | ||
|
|
||
| - HTTP/S Client URI endpoint identifier | ||
| - References the HTTP/S URI endpoint to which the Worker sends OpenTelemetry data. For example: `http://localhost:4319/v1/metrics`. | ||
| - The default identifier is `DESTINATION_OTEL_HTTP_CLIENT_URI`. | ||
| - HTTP/S Client TLS passphrase identifier (when TLS is enabled): | ||
| - The default identifier is `DESTINATION_OTEL_HTTP_CLIENT_KEY_PASS`. | ||
|
|
||
| {{% /tab %}} | ||
|
|
||
| {{% tab "Environment Variables" %}} | ||
|
|
||
| {{% observability_pipelines/configure_existing_pipelines/destination_env_vars/opentelemetry %}} | ||
|
|
||
| {{% /tab %}} | ||
| {{< /tabs >}} | ||
|
|
||
| ## Metrics | ||
|
|
||
| For [component metrics][6] and [destination buffer metrics][7] emitted by all destinations, see the [Pipelines Usage Metrics][8] documentation. To filter or group by OpenTelemetry destination metrics, use the tag `component_type:opentelemetry`. | ||
|
|
||
| ## How the destination works | ||
|
|
||
| ### Event batching | ||
|
|
||
| A batch of events is flushed when one of these conditions occurs. See [event batching][2] for more information. | ||
|
|
||
| | Maximum Events | Maximum Size (MB) | Timeout (seconds) | | ||
| |----------------|-------------------|---------------------| | ||
| | TKTK | TKTK | TKTK | | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Waiting for eng on this |
||
|
|
||
| [1]: https://app.datadoghq.com/observability-pipelines | ||
| [2]: /observability_pipelines/destinations/#event-batching | ||
| [3]: /observability_pipelines/configuration/set_up_pipelines/ | ||
| [4]: /api/latest/observability-pipelines/ | ||
| [5]: https://registry.terraform.io/providers/datadog/datadog/latest/docs/resources/observability_pipeline | ||
| [6]: /observability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics/#component-metrics | ||
| [7]: /observability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics/#destination-buffer-metrics | ||
| [8]: /observability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics/ | ||
| [9]: /observability_pipelines/processors/filter/ | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
5 changes: 5 additions & 0 deletions
5
...ty_pipelines/configure_existing_pipelines/destination_env_vars/opentelemetry.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| - HTTP/S client URI endpoint: | ||
| - The HTTP/S URI endpoint to which the Worker sends OpenTelemetry data. For example: `http://localhost:4319/v1/metrics`. | ||
| - The default environment variable is `DD_OP_DESTINATION_OTEL_HTTP_CLIENT_URI`. | ||
| - HTTP/S Client TLS passphrase (when enabled): | ||
| - The default environment variable is `DD_OP_DESTINATION_OTEL_HTTP_CLIENT_KEY_PASS`. |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Need to confirm with eng that the unsupported metric gets dropped and not the whole batch.