Improve the OpenTelemetry Collector dashboard

[cyrille-leclerc]

Changes

Fixes:

• [grafana] OpenTelemetry Collector Dashboard needs re-work #2735

Also add details to help:

• Understand the setup of the collector showing all the receivers, processors, and exporters
• Understand performance bottlenecks with per component Rate, error, and queue saturation indicators
• Identify failures and troubleshoot with:
  • Error indicators: both at a high level view through aggregated indicators and at the component level
  • Collector logs

Also integrate OTel Collector alerting based on the alerts defined in the otel operator helm chart here. Note that we can reuse the same alert rules on Docker Compose and VM deployments of the collector

Merge Requirements

For new features contributions, please make sure you have completed the following
essential items:

• CHANGELOG.md updated to document new feature additions
• Appropriate documentation updates in the docs
• Appropriate Helm chart updates in the helm-charts

Maintainers will not merge until the above have been completed. If you're unsure
which docs need to be changed ping the
@open-telemetry/demo-approvers.

[puckpuck]

@cyrille-leclerc do you know if this dashboard will also be published and maintained on Grafana's dashboard library for OpenTelemetry?

[cyrille-leclerc]

@cyrille-leclerc do you know if this dashboard will also be published and maintained on Grafana's dashboard library for OpenTelemetry?

It's my goal to publish this dashboard on Grafana's dashboard library for OpenTelemetry. I have small delay on this due to a problem uploading the dashboard.

Note that this dashboard is a bit specific to the OTel Demo because it's not only relying on Prometheus metrics but also on OpenSearch logs.
I'm considering producing in parallel a "promtheus-metrics-only" version of the dashboard putting a placeholder panel for the logs providing guidance on how to "bring your own logs datasource"