diff --git a/content/en/observability_pipelines/_index.md b/content/en/observability_pipelines/_index.md index 1518290ce85..48dfc59a97f 100644 --- a/content/en/observability_pipelines/_index.md +++ b/content/en/observability_pipelines/_index.md @@ -73,12 +73,12 @@ further_reading: text: Manage metric volume and tags in your environment with Observability Pipelines --- - +{{< tooltip text="term" tooltip="Additional information here" case="title" >}} ## Overview {{< img src="observability_pipelines/op_marketecture_06042025.png" alt="A graphic showing data being aggregated from a variety of sources, processed and enriched by the observability pipelines worker in your own environment, and then being routed to the security, analytics, and storage destinations of your choice" style="width:100%;" >}} -Datadog Observability Pipelines allows you to collect and process logs and metrics within your own infrastructure, and then route the data to different destinations. It gives you control over your observability data before it leaves your environment. +Datadog Observability Pipelines allows you to collect and process logs, metrics, and traces ({{< tooltip text="in Preview" tooltip="Traces for Observability Pipelines is in Preview. Contact your account manager to request access." >}}) within your own infrastructure, and then route the data to different destinations. It gives you control over your observability data before it leaves your environment. With out-of-the-box templates, you can build pipelines that redact sensitive data, enrich data, filter out noisy events, and route data to destinations like Datadog, SIEM tools, or cloud storage. diff --git a/content/en/observability_pipelines/configuration/_index.md b/content/en/observability_pipelines/configuration/_index.md index c5dda65a352..aac83f3a16d 100644 --- a/content/en/observability_pipelines/configuration/_index.md +++ b/content/en/observability_pipelines/configuration/_index.md @@ -20,7 +20,7 @@ further_reading: {{< img src="observability_pipelines/setup/pipeline_ui.png" alt="The pipelines page with a source going to two processors groups and two destinations" style="width:100%;" >}} -Observability Pipelines lets you collect and process logs and metrics within your own infrastructure, and then route them to different destinations. A pipeline consists of three core components: +Observability Pipelines lets you collect and process logs, metrics, and traces ({{< tooltip text="in Preview" tooltip="Traces for Observability Pipelines is in Preview. Contact your account manager to request access." >}}) within your own infrastructure, and then route them to different destinations. A pipeline consists of three core components: - [Source][1]: Receives data from a tool like the Datadog Agent. - [Processors][2]: Transform, enrich, or filter data. diff --git a/content/en/observability_pipelines/configuration/explore_templates.md b/content/en/observability_pipelines/configuration/explore_templates.md index d74a4a696d8..7e62e796032 100644 --- a/content/en/observability_pipelines/configuration/explore_templates.md +++ b/content/en/observability_pipelines/configuration/explore_templates.md @@ -75,6 +75,15 @@ Metrics capture signals about your environment and offer insight into your syste To help you manage the quality and volume of your metrics, use the Metric Tag Governance template to process them in Observability Pipelines before sending them to your destinations. You can use processors to keep only the metrics you need, standardize metrics tagging, and remove unwanted tags to prevent high cardinality. +{{% /tab %}} +{{% tab "Traces" %}} + +### Trace Sampling + +
Traces for Observability Pipelines is in Preview. Contact your account manager to request access.
+ +Ingesting and storing every trace can be costly at scale. Use the Trace Sampling template to ingest, process, and route your traces telemetry data to control costs while retaining the traces you need for troubleshooting and analysis. + {{% /tab %}} {{< /tabs >}} diff --git a/content/en/observability_pipelines/configuration/install_the_worker/_index.mdoc.md b/content/en/observability_pipelines/configuration/install_the_worker/_index.mdoc.md index d493094467a..d70e51b30e8 100644 --- a/content/en/observability_pipelines/configuration/install_the_worker/_index.mdoc.md +++ b/content/en/observability_pipelines/configuration/install_the_worker/_index.mdoc.md @@ -46,7 +46,7 @@ For RHEL and CentOS, the Observability Pipelines Worker supports versions 8.0 or {% /if %} -The Observability Pipelines Worker is software that runs in your environment to centrally aggregate and process your logs and metrics ({% glossary-tooltip term="preview" case="title" /%}), and then route them to different destinations. +The Observability Pipelines Worker is software that runs in your environment to centrally aggregate and process your logs, metrics, and traces ({{< tooltip text="in Preview" tooltip="Traces for Observability Pipelines is in Preview. Contact your account manager to request access." >}}) ({% glossary-tooltip term="preview" case="title" /%}), and then route them to different destinations. {% if equals($platform, "kubernetes") %} diff --git a/content/en/observability_pipelines/configuration/live_capture.md b/content/en/observability_pipelines/configuration/live_capture.md index 53b1d9048e1..6518c5ff4a5 100644 --- a/content/en/observability_pipelines/configuration/live_capture.md +++ b/content/en/observability_pipelines/configuration/live_capture.md @@ -14,6 +14,9 @@ products: - name: Metrics icon: metrics url: /observability_pipelines/configuration/?tab=metrics#pipeline-types +- name: Traces + icon: apm + url: /observability_pipelines/configuration/?tab=traces#pipeline-types --- {{< product-availability >}} diff --git a/content/en/observability_pipelines/configuration/set_up_pipelines.md b/content/en/observability_pipelines/configuration/set_up_pipelines.md index e8611215909..20deb3835ac 100644 --- a/content/en/observability_pipelines/configuration/set_up_pipelines.md +++ b/content/en/observability_pipelines/configuration/set_up_pipelines.md @@ -110,6 +110,33 @@ See [Export a Pipeline Configuration to JSON or Terraform][14] if you want to pr [6]: /observability_pipelines/configuration/pipeline_simulation/ [11]: /observability_pipelines/search_syntax/metrics/ +{{% /tab %}} +{{% tab "Traces" %}} + +
Traces for Observability Pipelines is in Preview. Contact your account manager to request access.
+ +1. Navigate to [Observability Pipelines][1]. +1. Select the [Trace Sampling][2] template. +1. Select and set up a [trace source][3]. +1. Select and set up [destinations][5] for your processed traces. +1. Click {{< ui >}}Edit{{< /ui >}} on a processor group to add, update, and validate your [processors][4] using [Pipeline Simulation][6]. + - **Notes**: + - For a pipeline canvas, there is a limit of 25 processor groups and a total of 150 processors. + - If you want to copy a processor, click the copy icon for that processor and then paste it (`Cmd+V` on Mac, `Ctrl+V` on Windows/Linux). + +#### Add another processor group + +{{< img src="observability_pipelines/setup/another_processor_group.png" alt="The Pipelines page showing two processor groups sending logs to the same destination" style="width:100%;" >}} + +{{% observability_pipelines/set_up_pipelines/add_another_processor_group %}} + +[1]: https://app.datadoghq.com/observability-pipelines +[2]: /observability_pipelines/configuration/explore_templates/?tab=traces#trace-sampling +[3]: /observability_pipelines/sources/?tab=traces#sources +[4]: /observability_pipelines/processors/?tab=traces#processors +[5]: /observability_pipelines/destinations/?tab=traces#destinations +[6]: /observability_pipelines/configuration/pipeline_simulation/ + {{% /tab %}} {{< /tabs >}} diff --git a/content/en/observability_pipelines/destinations/_index.md b/content/en/observability_pipelines/destinations/_index.md index df7a2e99197..551ec9c116a 100644 --- a/content/en/observability_pipelines/destinations/_index.md +++ b/content/en/observability_pipelines/destinations/_index.md @@ -9,7 +9,7 @@ further_reading: ## Overview -Use the Observability Pipelines Worker to send your processed logs and metrics to different destinations. Most Observability Pipelines destinations send events in batches to the downstream integration. See [Event batching](#event-batching) for more information. Some Observability Pipelines destinations also have fields that support template syntax, so you can set these fields based on specific fields. See [Template syntax](#template-syntax) for more information. +Use the Observability Pipelines Worker to send your processed logs, metrics, and traces ({{< tooltip text="in Preview" tooltip="Traces for Observability Pipelines is in Preview. Contact your account manager to request access." >}}) to different destinations. Most Observability Pipelines destinations send events in batches to the downstream integration. See [Event batching](#event-batching) for more information. Some Observability Pipelines destinations also have fields that support template syntax, so you can set these fields based on specific fields. See [Template syntax](#template-syntax) for more information. Select a destination in the left navigation menu to see more information about it. @@ -84,6 +84,14 @@ These are the available destinations: [3]: /observability_pipelines/destinations/http_client/ [4]: /observability_pipelines/destinations/splunk_hec/metrics +{{% /tab %}} +{{% tab "Traces" %}} + +
Traces for Observability Pipelines is in Preview. Contact your account manager to request access.
+ +- Datadog APM +- OpenTelemetry Traces + {{% /tab %}} {{< /tabs >}} diff --git a/content/en/observability_pipelines/processors/_index.md b/content/en/observability_pipelines/processors/_index.md index 0e5a2538d0e..14e5ed74ad5 100644 --- a/content/en/observability_pipelines/processors/_index.md +++ b/content/en/observability_pipelines/processors/_index.md @@ -14,7 +14,7 @@ further_reading:
The processors outlined in this documentation are specific to on-premises logging environments. To parse, structure, and enrich cloud-based logs, see the Log Management documentation.
-Use Observability Pipelines' processors to parse, structure, and enrich your logs and metrics. When you create a pipeline in the UI, pre-selected processors are added to your processor group based on the selected template. You can add additional processors and delete any existing ones based on your processing needs. +Use Observability Pipelines' processors to parse, structure, and enrich your logs, metrics, and traces ({{< tooltip text="in Preview" tooltip="Traces for Observability Pipelines is in Preview. Contact your account manager to request access." >}}). When you create a pipeline in the UI, pre-selected processors are added to your processor group based on the selected template. You can add additional processors and delete any existing ones based on your processing needs. Processor groups are executed from top to bottom. The order of the processors is important because events are checked by each processor, but only events that match the processor's filters are processed. To modify the order of the processors, use the drag handle on the top left corner of the processor you want to move. @@ -84,6 +84,21 @@ These are the available processors: [4]: /observability_pipelines/processors/tag_allow_block_list/ [5]: /observability_pipelines/processors/tag_cardinality_control/ +{{% /tab %}} +{{% tab "Traces" %}} + +
Traces for Observability Pipelines is in Preview. Contact your account manager to request access.
+ +- [Custom Processor][1] +- [Filter][2] +- [Sample][3] +- [Sensitive Data Scanner][4] + +[1]: /observability_pipelines/processors/custom_processor/ +[2]: /observability_pipelines/processors/filter/ +[3]: /observability_pipelines/processors/sample/ +[4]: /observability_pipelines/processors/sensitive_data_scanner/ + {{% /tab %}} {{< /tabs >}} diff --git a/content/en/observability_pipelines/sources/_index.md b/content/en/observability_pipelines/sources/_index.md index 499341f9bf7..d6864054367 100644 --- a/content/en/observability_pipelines/sources/_index.md +++ b/content/en/observability_pipelines/sources/_index.md @@ -85,6 +85,17 @@ These are the available sources: [1]: /observability_pipelines/sources/datadog_agent/ [2]: /observability_pipelines/sources/opentelemetry/ +{{% /tab %}} +{{% tab "Traces" %}} + +
Traces for Observability Pipelines is in Preview. Contact your account manager to request access.
+ +- [Datadog Agent][1] +- [OpenTelemetry][2] + +[1]: /observability_pipelines/sources/datadog_agent/ +[2]: /observability_pipelines/sources/opentelemetry/ + {{% /tab %}} {{< /tabs >}}