Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions content/en/observability_pipelines/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,12 +73,12 @@ further_reading:
text: Manage metric volume and tags in your environment with Observability Pipelines

---

{{< tooltip text="term" tooltip="Additional information here" case="title" >}}
## Overview

{{< img src="observability_pipelines/op_marketecture_06042025.png" alt="A graphic showing data being aggregated from a variety of sources, processed and enriched by the observability pipelines worker in your own environment, and then being routed to the security, analytics, and storage destinations of your choice" style="width:100%;" >}}

Datadog Observability Pipelines allows you to collect and process logs and metrics within your own infrastructure, and then route the data to different destinations. It gives you control over your observability data before it leaves your environment.
Datadog Observability Pipelines allows you to collect and process logs, metrics, and traces ({{< tooltip text="in Preview" tooltip="Traces for Observability Pipelines is in Preview. Contact your account manager to request access." >}}) within your own infrastructure, and then route the data to different destinations. It gives you control over your observability data before it leaves your environment.

With out-of-the-box templates, you can build pipelines that redact sensitive data, enrich data, filter out noisy events, and route data to destinations like Datadog, SIEM tools, or cloud storage.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ further_reading:

{{< img src="observability_pipelines/setup/pipeline_ui.png" alt="The pipelines page with a source going to two processors groups and two destinations" style="width:100%;" >}}

Observability Pipelines lets you collect and process logs and metrics within your own infrastructure, and then route them to different destinations. A pipeline consists of three core components:
Observability Pipelines lets you collect and process logs, metrics, and traces ({{< tooltip text="in Preview" tooltip="Traces for Observability Pipelines is in Preview. Contact your account manager to request access." >}}) within your own infrastructure, and then route them to different destinations. A pipeline consists of three core components:

- [Source][1]: Receives data from a tool like the Datadog Agent.
- [Processors][2]: Transform, enrich, or filter data.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,15 @@ Metrics capture signals about your environment and offer insight into your syste

To help you manage the quality and volume of your metrics, use the Metric Tag Governance template to process them in Observability Pipelines before sending them to your destinations. You can use processors to keep only the metrics you need, standardize metrics tagging, and remove unwanted tags to prevent high cardinality.

{{% /tab %}}
{{% tab "Traces" %}}

### Trace Sampling

<div class="alert alert-info">Traces for Observability Pipelines is in Preview. Contact your account manager to request access.</div>

Ingesting and storing every trace can be costly at scale. Use the Trace Sampling template to ingest, process, and route your traces telemetry data to control costs while retaining the traces you need for troubleshooting and analysis.

{{% /tab %}}
{{< /tabs >}}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ For RHEL and CentOS, the Observability Pipelines Worker supports versions 8.0 or

{% /if %}

The Observability Pipelines Worker is software that runs in your environment to centrally aggregate and process your logs and metrics ({% glossary-tooltip term="preview" case="title" /%}), and then route them to different destinations.
The Observability Pipelines Worker is software that runs in your environment to centrally aggregate and process your logs, metrics, and traces ({{< tooltip text="in Preview" tooltip="Traces for Observability Pipelines is in Preview. Contact your account manager to request access." >}}) ({% glossary-tooltip term="preview" case="title" /%}), and then route them to different destinations.

<!-- Kubernetes - Overview -->
{% if equals($platform, "kubernetes") %}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,9 @@ products:
- name: Metrics
icon: metrics
url: /observability_pipelines/configuration/?tab=metrics#pipeline-types
- name: Traces
icon: apm
url: /observability_pipelines/configuration/?tab=traces#pipeline-types
---

{{< product-availability >}}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,33 @@ See [Export a Pipeline Configuration to JSON or Terraform][14] if you want to pr
[6]: /observability_pipelines/configuration/pipeline_simulation/
[11]: /observability_pipelines/search_syntax/metrics/

{{% /tab %}}
{{% tab "Traces" %}}

<div class="alert alert-info">Traces for Observability Pipelines is in Preview. Contact your account manager to request access.</div>

1. Navigate to [Observability Pipelines][1].
1. Select the [Trace Sampling][2] template.
1. Select and set up a [trace source][3].
1. Select and set up [destinations][5] for your processed traces.
1. Click {{< ui >}}Edit{{< /ui >}} on a processor group to add, update, and validate your [processors][4] using [Pipeline Simulation][6].
- **Notes**:
- For a pipeline canvas, there is a limit of 25 processor groups and a total of 150 processors.
- If you want to copy a processor, click the copy icon for that processor and then paste it (`Cmd+V` on Mac, `Ctrl+V` on Windows/Linux).

#### Add another processor group

{{< img src="observability_pipelines/setup/another_processor_group.png" alt="The Pipelines page showing two processor groups sending logs to the same destination" style="width:100%;" >}}

{{% observability_pipelines/set_up_pipelines/add_another_processor_group %}}

[1]: https://app.datadoghq.com/observability-pipelines
[2]: /observability_pipelines/configuration/explore_templates/?tab=traces#trace-sampling
[3]: /observability_pipelines/sources/?tab=traces#sources
[4]: /observability_pipelines/processors/?tab=traces#processors
[5]: /observability_pipelines/destinations/?tab=traces#destinations
[6]: /observability_pipelines/configuration/pipeline_simulation/

{{% /tab %}}
{{< /tabs >}}

Expand Down
10 changes: 9 additions & 1 deletion content/en/observability_pipelines/destinations/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ further_reading:

## Overview

Use the Observability Pipelines Worker to send your processed logs and metrics to different destinations. Most Observability Pipelines destinations send events in batches to the downstream integration. See [Event batching](#event-batching) for more information. Some Observability Pipelines destinations also have fields that support template syntax, so you can set these fields based on specific fields. See [Template syntax](#template-syntax) for more information.
Use the Observability Pipelines Worker to send your processed logs, metrics, and traces ({{< tooltip text="in Preview" tooltip="Traces for Observability Pipelines is in Preview. Contact your account manager to request access." >}}) to different destinations. Most Observability Pipelines destinations send events in batches to the downstream integration. See [Event batching](#event-batching) for more information. Some Observability Pipelines destinations also have fields that support template syntax, so you can set these fields based on specific fields. See [Template syntax](#template-syntax) for more information.

Select a destination in the left navigation menu to see more information about it.

Expand Down Expand Up @@ -84,6 +84,14 @@ These are the available destinations:
[3]: /observability_pipelines/destinations/http_client/
[4]: /observability_pipelines/destinations/splunk_hec/metrics

{{% /tab %}}
{{% tab "Traces" %}}

<div class="alert alert-info">Traces for Observability Pipelines is in Preview. Contact your account manager to request access.</div>

- Datadog APM
- OpenTelemetry Traces

{{% /tab %}}
{{< /tabs >}}

Expand Down
17 changes: 16 additions & 1 deletion content/en/observability_pipelines/processors/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ further_reading:

<div class="alert alert-info">The processors outlined in this documentation are specific to on-premises logging environments. To parse, structure, and enrich cloud-based logs, see the <a href="https://docs.datadoghq.com/logs/log_configuration/logs_to_metrics">Log Management</a> documentation.</div>

Use Observability Pipelines' processors to parse, structure, and enrich your logs and metrics. When you create a pipeline in the UI, pre-selected processors are added to your processor group based on the selected template. You can add additional processors and delete any existing ones based on your processing needs.
Use Observability Pipelines' processors to parse, structure, and enrich your logs, metrics, and traces ({{< tooltip text="in Preview" tooltip="Traces for Observability Pipelines is in Preview. Contact your account manager to request access." >}}). When you create a pipeline in the UI, pre-selected processors are added to your processor group based on the selected template. You can add additional processors and delete any existing ones based on your processing needs.

Processor groups are executed from top to bottom. The order of the processors is important because events are checked by each processor, but only events that match the processor's filters are processed. To modify the order of the processors, use the drag handle on the top left corner of the processor you want to move.

Expand Down Expand Up @@ -84,6 +84,21 @@ These are the available processors:
[4]: /observability_pipelines/processors/tag_allow_block_list/
[5]: /observability_pipelines/processors/tag_cardinality_control/

{{% /tab %}}
{{% tab "Traces" %}}

<div class="alert alert-info">Traces for Observability Pipelines is in Preview. Contact your account manager to request access.</div>

- [Custom Processor][1]
- [Filter][2]
- [Sample][3]
- [Sensitive Data Scanner][4]

[1]: /observability_pipelines/processors/custom_processor/
[2]: /observability_pipelines/processors/filter/
[3]: /observability_pipelines/processors/sample/
[4]: /observability_pipelines/processors/sensitive_data_scanner/

{{% /tab %}}
{{< /tabs >}}

Expand Down
11 changes: 11 additions & 0 deletions content/en/observability_pipelines/sources/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,17 @@ These are the available sources:
[1]: /observability_pipelines/sources/datadog_agent/
[2]: /observability_pipelines/sources/opentelemetry/

{{% /tab %}}
{{% tab "Traces" %}}

<div class="alert alert-info">Traces for Observability Pipelines is in Preview. Contact your account manager to request access.</div>

- [Datadog Agent][1]
- [OpenTelemetry][2]

[1]: /observability_pipelines/sources/datadog_agent/
[2]: /observability_pipelines/sources/opentelemetry/

{{% /tab %}}
{{< /tabs >}}

Expand Down
Loading