[OTAGENT-512] Add more configs and examples to otel agent gateway

[songy23]

What does this PR do?

Add config FeatureGates to otel agent gateway matching the implementation in helm charts: http://github.com/DataDog/helm-charts/blob/main/charts/datadog/templates/otel-agent-gateway-deployment.yaml.

The following configs are already supported with override, so no need to add them again to otel agent gateway feature:

1. Replicas
2. NodeSelector
3. Affinity
4. Tolerations
5. PriorityClassName
6. Env
7. EnvFrom
8. Image

examples/datadogagent/datadog-agent-with-otelagentgateway.yaml shows an example of setting all the fields above (and many others) for otel agent gateway.

Motivation

Implement otel agent gateway in operator.

Additional Notes

Anything else we should know when reviewing?

Minimum Agent Versions

Are there minimum versions of the Datadog Agent and/or Cluster Agent required?

• Agent: v7.74.0

Describe your test plan

For QA, use examples/datadogagent/datadog-agent-with-otelagentgateway.yaml. You can uncomment / tweak the parameters based on your needs. Verify the node selector, affinity, labels, replicas etc. in the otel agent gateway deployment and pods are expected.

Checklist

• PR has at least one valid label: bug, enhancement, refactoring, documentation, tooling, and/or dependencies
• PR has a milestone or the qa/skip-qa label
• All commits are signed (see: signing commits)

[codecov-commenter]

Codecov Report

❌ Patch coverage is 42.10526% with 11 lines in your changes missing coverage. Please review.
✅ Project coverage is 37.88%. Comparing base (dfeb5ee) to head (9779963).

Files with missing lines	Patch %	Lines
pkg/testutils/builder.go	0.00%	6 Missing ⚠️
...datadogagent/component/otelagentgateway/default.go	0.00%	5 Missing ⚠️

❌ Your patch status has failed because the patch coverage (42.10%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2490      +/-   ##
==========================================
+ Coverage   37.86%   37.88%   +0.02%     
==========================================
  Files         303      303              
  Lines       25995    26002       +7     
==========================================
+ Hits         9843     9851       +8     
+ Misses      15409    15408       -1     
  Partials      743      743              

Flag	Coverage Δ
unittests	37.88% <42.10%> (+0.02%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...r/datadogagent/feature/otelagentgateway/feature.go	90.97% <100.00%> (+0.57%)	⬆️
...datadogagent/component/otelagentgateway/default.go	0.00% <0.00%> (ø)
pkg/testutils/builder.go	0.00% <0.00%> (ø)

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update dfeb5ee...9779963. Read the comment docs.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[joepeeples]

Changing the description to "feature gates" would be more readable, but feel free to ignore if there are technical reasons to keep FeatureGates. Thanks!

Approved by #agent-onboarding while Timothée is OOO

[songy23]

Test failure seems to be from mainline http://gitlab.ddbuild.io/DataDog/datadog-operator/-/jobs/1366828575