Add documentation for Elastic Agent as an OTel Collector#5545
Add documentation for Elastic Agent as an OTel Collector#5545vishaangelova wants to merge 2 commits intomainfrom
Conversation
✅ Vale Linting ResultsNo issues found on modified lines! The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale. |
🔍 Preview links for changed docs |
6db2445 to
d2d234c
Compare
| endpoints: | ||
| - https://example.com | ||
|
|
||
| exporters: |
There was a problem hiding this comment.
Should the exporter section include an elasticsearch exporter if the output section already includes the ES output? For this configuration example I used the snippet from https://github.com/elastic/elastic-agent/blob/main/docs/hybrid-agent-beats-receivers.md#hybrid-agent.
|
|
||
| Data collected by a Beat receiver is written by the {{es}} exporter. As with traditional {{agent}} and {{beats}}, this data is processed by ingest pipelines, schematized, and stored in the appropriate data stream. | ||
|
|
||
| Data collected by an OTel-native receiver follows OpenTelemetry semantic conventions. When this data arrives at the Elastic cluster, it bypasses ingest pipelines and is stored directly in an OTel-specific data stream. |
There was a problem hiding this comment.
@nimarezainia is this statement correct? You had a comment in your draft google doc which said this might need clarification.
| A standalone hybrid {{agent}} can enroll into {{fleet}} in the field if an in-place upgrade to {{fleet}}-managed is required. The standalone EDOT Collector does not support enrollment into {{fleet}}. | ||
| ::: | ||
|
|
||
| ## Frequently asked questions [faq] |
There was a problem hiding this comment.
Should we rather move this section to a separate doc positioned under this one? I think keeping the FAQ here makes the Q&A pairs more visible.
There was a problem hiding this comment.
I'd rather not have an FAQ as a pattern unless we deem it strictly necessary.
theletterf
left a comment
There was a problem hiding this comment.
Great work! First review pass.
| --- | ||
|
|
||
| # {{agent}} as an OpenTelemetry Collector [elastic-agent-otel-collector] | ||
|
|
There was a problem hiding this comment.
Guess you thought about this, but... Section-level applies to?
|
|
||
| # {{agent}} as an OpenTelemetry Collector [elastic-agent-otel-collector] | ||
|
|
||
| Starting with version 9.3, {{agent}} runs on the [Elastic Distribution of OpenTelemetry (EDOT) Collector](elastic-agent://reference/edot-collector/index.md) internally. Rather than managing separate {{beats}} sub-processes, the agent now runs data collection inside an embedded OpenTelemetry (OTel) Collector process, leveraging the extensibility and interoperability of the OTel ecosystem. This new architecture brings OTel capabilities while maintaining compatibility with existing {{beats}}-based workflows and integrations. |
There was a problem hiding this comment.
| Starting with version 9.3, {{agent}} runs on the [Elastic Distribution of OpenTelemetry (EDOT) Collector](elastic-agent://reference/edot-collector/index.md) internally. Rather than managing separate {{beats}} sub-processes, the agent now runs data collection inside an embedded OpenTelemetry (OTel) Collector process, leveraging the extensibility and interoperability of the OTel ecosystem. This new architecture brings OTel capabilities while maintaining compatibility with existing {{beats}}-based workflows and integrations. | |
| Starting with version 9.3, {{agent}} runs the [Elastic Distribution of OpenTelemetry (EDOT) Collector](elastic-agent://reference/edot-collector/index.md). Rather than managing separate {{beats}} sub-processes, the agent collects data through an embedded OpenTelemetry (OTel) Collector process, leveraging the extensibility and interoperability of the OTel ecosystem. This architecture brings OTel capabilities while maintaining compatibility with existing {{beats}}-based workflows and integrations. |
| Previously {{agent}} acted as a supervisor that launched and managed individual {{beats}} processes to collect telemetry ({{filebeat}}, {{metricbeat}}, and others), each running as a separate sub-process. | ||
|
|
||
| With the new {{agent}} OTel Collector architecture: |
There was a problem hiding this comment.
I would focus just on the present. I'm not sure if the past architecture is relevant.
| With the new {{agent}} OTel Collector architecture: | ||
|
|
||
| * {{agent}} embeds the EDOT Collector as its runtime, eliminating the overhead associated with managing separate sub-processes. | ||
| * Instead of running as individual {{beats}} processes, Beat inputs can run inside the EDOT Collector as [Beat receivers](#beat-receivers). |
There was a problem hiding this comment.
| * Instead of running as individual {{beats}} processes, Beat inputs can run inside the EDOT Collector as [Beat receivers](#beat-receivers). | |
| * Instead of running as individual {{beats}} processes, Beats inputs can run inside the EDOT Collector as [Beat receivers](#beat-receivers). |
Beat can be used without 's'?
| :alt: Diagram showing data ingestion with Beat receivers and OTel receivers as part of an OTel Collector | ||
| ::: | ||
|
|
||
| :::{note} |
There was a problem hiding this comment.
Applies_to for admonitions?
|
|
||
| ## Beat receivers [beat-receivers] | ||
|
|
||
| A _Beat receiver_ is a Beat input and its associated processors, wrapped to run as an OTel receiver inside the EDOT Collector. Beat receivers produce the exact same data, formatted according to the [Elastic Common Schema](ecs://reference/index.md) (ECS), as current Beat inputs — they do not output data in the OTLP schema. |
There was a problem hiding this comment.
| A _Beat receiver_ is a Beat input and its associated processors, wrapped to run as an OTel receiver inside the EDOT Collector. Beat receivers produce the exact same data, formatted according to the [Elastic Common Schema](ecs://reference/index.md) (ECS), as current Beat inputs — they do not output data in the OTLP schema. | |
| A _Beat receiver_ is a Beat input and its associated processors, wrapped to run as an OTel receiver inside the EDOT Collector. Beat receivers produce the exact same data, formatted according to the [Elastic Common Schema](ecs://reference/index.md) (ECS), as current Beat inputs. Beat receivers don't output data in the OTLP schema. |
| :alt: Beat input configuration translated into an OTel configuration | ||
| ::: | ||
|
|
||
| ## {{agent}} as a hybrid agent [hybrid-elastic-agent] |
There was a problem hiding this comment.
This in my reads different than "EA as an OTel Collector". Could we make it more compatible, like "Hybrid data collection"?
| A standalone hybrid {{agent}} can enroll into {{fleet}} in the field if an in-place upgrade to {{fleet}}-managed is required. The standalone EDOT Collector does not support enrollment into {{fleet}}. | ||
| ::: | ||
|
|
||
| ## Frequently asked questions [faq] |
There was a problem hiding this comment.
I'd rather not have an FAQ as a pattern unless we deem it strictly necessary.
Summary
This PR adds a new “Elastic Agent as an OTel Collector” document with details about:
Resolves elastic/ingest-docs#1882
Generative AI disclosure
Tool(s) and model(s) used: Cursor with claude-4.5-sonnet and Auto models