telemetry: add OTLP traces pipeline

[samcm]

Upstreams blob-devnets' traces pipeline so devnets stamped from the template get traces out of the box: Vector gains an otlp_in source (gRPC :4317, HTTP :4318) and an otlp_traces sink to the prod analytics OTLP gateway, geth exports RPC spans via --rpc.telemetry, and lighthouse via --telemetry-collector-url. No new sops keys needed — the traces sink reuses secret_loki.password already present for the logs sink. The bootnode CL here is teku, which has no equivalent telemetry flag, so only its geth is wired up.

The second commit wires three more clients: reth via --tracing-otlp (HTTP :4318; filter=info because the default debug produced ~50GB of spans during a sync, reth #21608; OTEL_RESOURCE_ATTRIBUTES carries the instance id since reth hardcodes service.name=reth), prysm via --enable-tracing (HTTP :4318 only — the endpoint must be set explicitly since the flag default is a stale Jaeger URL; sample fraction raised from the 0.20 default to 1.0), and grandine via --telemetry-metrics-url (gRPC :4317 only — despite the name it exports tracing spans, grandine #487).

Support matrix

Client	Status
geth, lighthouse, reth, prysm, grandine	wired
besu	skipped — switching to OTLP push would remove the Prometheus scrape endpoint
teku, lodestar, nethermind	skipped — need image or fork work; only generic spans available
erigon, ethrex, nimbus	skipped — no OTel support upstream

Version floors: reth ≥1.9.0 (--tracing-otlp.sample-ratio ≥1.10.0), prysm ≥5.3.1, grandine ≥2.0.2. This inventory pins reth/prysm to upstream :latest and grandine to ethpandaops/grandine:develop, all comfortably past the floors; an unknown flag fails loudly at container start either way.

[qu0b-reviewer]

🤖 qu0b-reviewer

Summary

This PR adds an OTLP traces pipeline to the devnet infrastructure: a Vector opentelemetry source receives traces from EL/CL clients and forwards them to the analytics gateway. Geth, reth, prysm, lighthouse, and grandine are each configured to push OTLP spans to the local Vector instance.

Issues

• 🟡 ansible/inventories/devnet-0/group_vars/bootnode.yaml:107–109 — Bootnode geth gets --rpc.telemetry, --rpc.telemetry.endpoint, and --rpc.telemetry.instance-id but is missing --rpc.telemetry.sample-ratio=1. The full geth config (geth.yaml:30) includes this flag with an explicit comment that it's "Required until geth ships the fix for the SampleRatio default" (linked to ethereum/go-ethereum#34948). Without it the bootnode's geth uses the default sample ratio — which, per that note, means zero traces emitted. Three lines of dead config on every bootnode. Either add --rpc.telemetry.sample-ratio=1 to the bootnode's list, or drop the 3 OTLP args if bootnode tracing was intentionally excluded.

Suggestions

• ansible/inventories/devnet-0/group_vars/bootnode.yaml — Consider also adding --rpc.telemetry.tags=... (mirroring geth.yaml:31) so bootnode traces carry execution_client/consensus_client/etc. metadata, matching the full-node config.

---

_{Reviewed @ 8f68d188
"The forest has ears, the field has eyes." — Hungarian proverb}