feat(monitoring): opt-in gateway distributed tracing + Tempo in telemetry playground

CHANGELOG.md

@@ -8,6 +8,7 @@
 ### Major Features
 - **io_uring opt-in feature gate**: A new `IOUring` feature gate (`spec.featureGates.IOUring: true`) enables PostgreSQL 18 asynchronous I/O (`io_method=io_uring`). Because the container runtime's default seccomp profile strips the `io_uring_setup/enter/register` syscalls, the operator also relaxes the postgres container seccomp profile — pointing the pods at a hardened Localhost profile that re-allows only those three syscalls — when the gate is on, so no external Kyverno policy is needed. It is **opt-in** (default off) since io_uring relaxes the sandbox. The Localhost profile path is operator-level config via the Helm value `operator.ioUring.seccompProfile` (default `profiles/documentdb-iouring.json`). See [io_uring documentation](docs/operator-public-documentation/io-uring.md) and the [feature playground](documentdb-playground/io-uring-feature/).
 - **Gateway OTLP metrics in the per-pod sidecar**: when `spec.monitoring.enabled=true`, the OTel Collector sidecar now exposes an OTLP/gRPC receiver on `127.0.0.1:4317` and the documentdb-gateway is configured (via `OTEL_EXPORTER_OTLP_ENDPOINT` and `OTEL_METRICS_ENABLED`) to push its `db_client_*` metrics there. The sidecar's existing prometheus exporter re-exports them alongside the existing `documentdb.postgres.up` sqlquery output, with per-pod attribution added by the collector's resource processor. No new CRD fields; this turns on automatically wherever monitoring was already enabled.
+- **Gateway distributed tracing (opt-in)**: a new `spec.monitoring.tracing` stanza (`enabled`, `otlpEndpoint`, `sqlCommenterEnabled`) turns on the gateway's OTLP trace exporter (`OTEL_TRACES_ENABLED`) and adds a `traces` pipeline to the per-pod OTel Collector sidecar that pushes spans (OTLP/gRPC) to a central trace backend such as Tempo. Unlike metrics — which Prometheus scrapes from each sidecar — traces must be pushed, so `otlpEndpoint` is required. With `sqlCommenterEnabled`, the gateway injects a W3C `traceparent` SQL comment into its Postgres queries for trace/log correlation. Requires a gateway image built with trace support (gateway ≥ 0.115).
 - **Two-Phase Extension Upgrade**: New `spec.schemaVersion` field separates binary upgrades (`spec.documentDBVersion`) from irreversible schema migrations (`ALTER EXTENSION UPDATE`). The default behavior gives you a rollback-safe window — update the binary first, validate, then finalize the schema. Set `schemaVersion: "auto"` for single-step upgrades in development environments. See the [upgrade guide](docs/operator-public-documentation/preview/operations/upgrades.md) for details.
 
 ### Breaking Changes
@@ -18,6 +19,7 @@
 ### Playground & Examples
 - **io_uring feature playground**: New `documentdb-playground/io-uring-feature/` demonstrates the operator-native `IOUring` opt-in, modeled on the upstream cnpg-playground seccomp approach — a kind cluster that `extraMount`s the curated Localhost seccomp profile, a DaemonSet installer for real clusters, the DocumentDB CR with `spec.featureGates.IOUring: true`, and verification steps. No Kyverno policy is required.
 - **Container metrics reference collector**: The telemetry playground now includes a reference OpenTelemetry Collector DaemonSet under `documentdb-playground/telemetry/container-metrics/` for clusters that do not already collect kubelet-backed container metrics. It scrapes each node's local kubelet for container, pod, and node CPU/memory/network/filesystem metrics and exposes them via Prometheus. The production operator chart does not install this platform-level collector; tenant DocumentDB clusters do not receive kubelet privileges.
+- **Tracing in the local telemetry playground**: `documentdb-playground/telemetry/local/` now deploys a Tempo trace backend in the `observability` namespace, enables `spec.monitoring.tracing` on the demo cluster, and provisions a Grafana Tempo datasource with trace→metrics correlation. `validate.sh` checks Tempo readiness and gateway spans; `deploy.sh` accepts `DOCUMENTDB_VERSION` to pull a tracing-capable gateway image.
 
 ### Testing infrastructure
 - **Unified E2E test suite ([#346](https://github.com/documentdb/documentdb-kubernetes-operator/pull/346))**: The four legacy end-to-end workflows (`test-integration.yml`, `test-E2E.yml`, `test-backup-and-restore.yml`, `test-upgrade-and-rollback.yml`) and their bash / JavaScript (mongosh) / Python (pymongo) glue have been replaced by a single Go / Ginkgo v2 / Gomega suite under `test/e2e/`. Specs are organised by CRD operation (lifecycle, scale, data, performance, backup, tls, feature gates, exposure, status, upgrade), reuse CloudNative-PG's `tests/utils` packages as a library, and speak the Mongo wire protocol via `go.mongodb.org/mongo-driver/v2`.

documentdb-playground/telemetry/local/README.md

@@ -1,6 +1,6 @@
 # DocumentDB Telemetry Playground (Local)
 
-A metrics-focused observability stack for DocumentDB on a local Kind cluster. Deploys a 3-instance HA cluster with the in-pod OTel sidecar enabled, a reference container-metrics DaemonSet, and a pre-configured Grafana dashboard for container/pod resource metrics.
+An observability stack for DocumentDB on a local Kind cluster. Deploys a 3-instance HA cluster with the in-pod OTel sidecar enabled, a reference container-metrics DaemonSet, **distributed tracing** (gateway spans → Tempo), and pre-configured Grafana dashboards + datasources for metrics and traces.
 
 ## Prerequisites
 
@@ -26,6 +26,10 @@ kubectl port-forward svc/grafana 3000:3000 -n observability --context kind-docum
 kubectl port-forward svc/prometheus 9090:9090 -n observability --context kind-documentdb-telemetry
 # → http://localhost:9090
 
+# 3b. Open Tempo directly (optional — Grafana already queries it)
+kubectl port-forward svc/tempo 3200:3200 -n observability --context kind-documentdb-telemetry
+# → http://localhost:3200
+
 # 4. Validate data is flowing
 ./scripts/validate.sh
 ```
@@ -42,13 +46,14 @@ The operator chart is installed **from this branch** (`operator/documentdb-helm-
 | cert-manager | `cert-manager` | TLS certificate management |
 | DocumentDB operator | `documentdb-operator` | Operator + CNPG (Helm chart from this branch) |
 | DocumentDB HA cluster | `documentdb-preview-ns` | 1 primary + 2 streaming replicas |
-| OTel Collector sidecar | `documentdb-preview-ns` | One per pod, injected by the operator's CNPG sidecar plugin when `spec.monitoring.enabled=true`. Receives gateway OTLP metrics and runs the `sqlquery` Postgres-health receiver. |
+| OTel Collector sidecar | `documentdb-preview-ns` | One per pod, injected by the operator's CNPG sidecar plugin when `spec.monitoring.enabled=true`. Receives gateway OTLP metrics **and traces** and runs the `sqlquery` Postgres-health receiver. |
 | Container-metrics DaemonSet | `documentdb-operator` | One OTel Collector per node from `documentdb-playground/telemetry/container-metrics/`. Scrapes each node's local kubelet for container/pod/node CPU, memory, network, filesystem metrics. |
 | Prometheus | `observability` | Metrics storage + alerting rules; scrapes the per-pod sidecar and reference DaemonSet via annotation discovery |
-| Grafana | `observability` | Dashboard (Internals — container/pod resources) |
+| Tempo | `observability` | Trace storage. Receives OTLP spans pushed by the per-pod sidecars' traces pipeline and serves them to Grafana. Enabled by `spec.monitoring.tracing`. |
+| Grafana | `observability` | Dashboards + Prometheus/Tempo datasources with trace→metrics correlation |
 | Traffic generators | `documentdb-preview-ns` | Read/write workload via mongosh |
 
-There is **no central OTel Collector Deployment**. Per-pod sidecars handle pod-local signals (Postgres health, gateway OTLP); the playground's reference DaemonSet handles node-local kubelet scraping for container resource metrics.
+There is **no central OTel Collector Deployment**. Per-pod sidecars handle pod-local signals (Postgres health, gateway OTLP metrics/traces); the playground's reference DaemonSet handles node-local kubelet scraping for container resource metrics. Metrics are *scraped* by Prometheus from each sidecar; traces are *pushed* (OTLP) from each sidecar to the central Tempo backend, because traces cannot be scraped.
 
 ## Architecture
 
@@ -57,15 +62,17 @@ graph TB
     subgraph cluster["Kind Cluster (documentdb-telemetry)"]
         subgraph obs["observability namespace"]
             prometheus["Prometheus<br/>annotation discovery"]
-            grafana["Grafana<br/>:3000<br/>Internals dashboard"]
+            tempo["Tempo<br/>:3200 (query) / :4317 (OTLP)<br/>traces store"]
+            grafana["Grafana<br/>:3000<br/>Prometheus + Tempo"]
             prometheus --> grafana
+            tempo --> grafana
         end
 
         subgraph docdb["documentdb-preview-ns"]
             subgraph pod1["Pod: preview-1 (primary)"]
                 pg1["postgres :5432"]
                 gw1["documentdb-gateway :10260"]
-                otel1["otel-collector sidecar<br/>sqlquery + otlp / Prom :9188"]
+                otel1["otel-collector sidecar<br/>sqlquery + otlp / Prom :9188<br/>traces pipeline"]
             end
             subgraph pod2["Pod: preview-2 (replica)"]
                 pg2["postgres :5432"]
@@ -92,6 +99,9 @@ graph TB
         prometheus -- "scrape :9188 (annotation)" --> otel2
         prometheus -- "scrape :9188 (annotation)" --> otel3
         prometheus -- "scrape :8889 (annotation)" --> ds
+        otel1 -- "push spans OTLP :4317" --> tempo
+        otel2 -- "push spans OTLP :4317" --> tempo
+        otel3 -- "push spans OTLP :4317" --> tempo
     end
 
     user["Browser"] --> grafana
@@ -107,8 +117,8 @@ local/
 │   ├── validate.sh            # Health check — verifies sidecar + data flow
 │   └── teardown.sh            # Deletes cluster and proxy containers
 ├── k8s/
-│   ├── observability/         # Namespace, Prometheus (annotation discovery), Grafana
-│   ├── documentdb/            # DocumentDB CR (with spec.monitoring.enabled) + credentials
+│   ├── observability/         # Namespace, Prometheus (annotation discovery), Grafana, Tempo (traces)
+│   ├── documentdb/            # DocumentDB CR (spec.monitoring.enabled + spec.monitoring.tracing) + credentials
 │   └── traffic/               # Traffic generator services + jobs
 └── dashboards/
     └── internals.json         # Container & pod resources dashboard (DaemonSet kubeletstats)
@@ -126,6 +136,54 @@ One dashboard is auto-provisioned in the **DocumentDB** folder:
 
 Dashboards auto-refresh every 30 seconds. Edits made in the Grafana UI persist until the pod restarts.
 
+## Tracing
+
+Distributed tracing is enabled via `spec.monitoring.tracing` on the DocumentDB CR
+(`k8s/documentdb/cluster.yaml`):
+
+```yaml
+monitoring:
+  enabled: true
+  exporter:
+    prometheus:
+      port: 9188
+  tracing:
+    enabled: true
+    otlpEndpoint: tempo.observability.svc.cluster.local:4317
+    sqlCommenterEnabled: true
+```
+
+When enabled, the operator:
+
+1. Sets `OTEL_TRACES_ENABLED=true` (and, with `sqlCommenterEnabled`,
+   `DOCUMENTDB_SQL_COMMENTER_ENABLED=true`) on the gateway container so it exports
+   OTLP spans to the co-located sidecar.
+2. Adds a **traces pipeline** to the sidecar collector (`otlp` receiver →
+   `resource`/`batch` → `otlp/traces` exporter) that pushes spans to **Tempo**.
+
+Unlike metrics — which Prometheus *scrapes* from each sidecar — traces are
+*pushed* to the central Tempo backend, since traces cannot be scraped.
+
+**View traces:** Grafana → **Explore → Tempo**, search `service.name = documentdb_gateway`.
+A single request produces `gateway.request → gateway.process_request →
+postgres.*` spans. With `sqlCommenterEnabled`, gateway-issued Postgres queries
+carry a `/* traceparent=… */` comment so database logs correlate to the trace.
+
+> **Gateway image requirement:** tracing requires a gateway built with OTLP
+> trace support (the feature landed in gateway **0.115**). The chart's default
+> gateway version predates it. Set `DOCUMENTDB_VERSION` to a tracing-capable
+> version when deploying, e.g.:
+>
+> ```bash
+> DOCUMENTDB_VERSION=0.115.0 ./scripts/deploy.sh
+> ```
+>
+> With a pre-0.115 gateway the pipeline is wired correctly but no spans are
+> produced (`validate.sh` reports this as a warning, not a failure). For
+> client → gateway → Postgres trace linking, use a fully OTel-instrumented
+> client that passes a W3C `traceparent` in the Mongo `comment` field (the
+> mongosh-based traffic generators here do not emit client spans).
+
 ## Alerting Rules
 
 Prometheus includes a sample alerting rule:

documentdb-playground/telemetry/local/k8s/documentdb/cluster.yaml

@@ -29,7 +29,8 @@ spec:
   resource:
     storage:
       pvcSize: 5Gi
-  sidecarInjectorPluginName: cnpg-i-sidecar-injector.documentdb.io
+  plugins:
+    sidecarInjectorName: cnpg-i-sidecar-injector.documentdb.io
   # Enable the OTel Collector sidecar (one per pod). The sidecar:
   #   - receives OTLP metrics from the documentdb-gateway on localhost:4317
   #   - exposes Prometheus /metrics on the configured port
@@ -43,3 +44,17 @@ spec:
     exporter:
       prometheus:
         port: 9188
+    # Distributed tracing (opt-in). The gateway exports OTLP spans to the
+    # co-located sidecar, which forwards them to the central Tempo backend in
+    # the observability namespace. Unlike metrics (scraped by Prometheus),
+    # traces must be pushed to a trace store.
+    #
+    # NOTE: requires a gateway image built with tracing support (>= 0.115).
+    # The operator's default gateway image predates this; override it via the
+    # chart value image.gateway (or spec.image.gateway) when deploying.
+    tracing:
+      enabled: true
+      otlpEndpoint: tempo.observability.svc.cluster.local:4317
+      # Inject a W3C traceparent SQL comment into gateway-issued Postgres
+      # queries so PostgreSQL logs correlate to gateway traces.
+      sqlCommenterEnabled: true

documentdb-playground/telemetry/local/k8s/observability/grafana.yaml

@@ -65,6 +65,19 @@ data:
         access: proxy
         url: http://prometheus.observability.svc:9090
         isDefault: true
+      - name: Tempo
+        type: tempo
+        uid: tempo
+        access: proxy
+        url: http://tempo.observability.svc:3200
+        jsonData:
+          # Trace -> metrics correlation via Tempo-generated span metrics.
+          tracesToMetrics:
+            datasourceUid: prometheus
+          serviceMap:
+            datasourceUid: prometheus
+          nodeGraph:
+            enabled: true
 ---
 apiVersion: v1
 kind: ConfigMap

documentdb-playground/telemetry/local/k8s/observability/tempo.yaml

@@ -0,0 +1,146 @@
+# ============================================================
+# Tempo - Distributed tracing backend (traces store)
+# ⚠️  DEMO/PLAYGROUND ONLY — single-binary, local ephemeral
+# storage (emptyDir), no auth. Not for production use.
+# ============================================================
+#
+# Why Tempo (and not Prometheus) for traces:
+#   Metrics are *scraped* by Prometheus from each per-pod OTel sidecar.
+#   Traces cannot be scraped — each sidecar *pushes* spans (OTLP/gRPC) to a
+#   central store. Tempo receives OTLP on :4317 and Grafana queries it on :3200.
+#
+# The DocumentDB CR points the sidecar traces pipeline at
+#   tempo.observability.svc.cluster.local:4317
+# via spec.monitoring.tracing.otlpEndpoint.
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: tempo-config
+  namespace: observability
+data:
+  tempo.yaml: |
+    server:
+      http_listen_port: 3200
+      grpc_listen_port: 9096
+    distributor:
+      receivers:
+        otlp:
+          protocols:
+            grpc:
+              endpoint: 0.0.0.0:4317
+            http:
+              endpoint: 0.0.0.0:4318
+    ingester:
+      max_block_duration: 5m
+    compactor:
+      compaction:
+        block_retention: 1h
+    storage:
+      trace:
+        backend: local
+        local:
+          path: /var/tempo/blocks
+        wal:
+          path: /var/tempo/wal
+    # Generate service-graph and span metrics so Grafana's trace->metrics and
+    # node-graph views work. Metrics are exposed for Prometheus to scrape.
+    metrics_generator:
+      registry:
+        external_labels:
+          source: tempo
+      storage:
+        path: /var/tempo/generator/wal
+      processor:
+        local_blocks:
+          filter_server_spans: false
+    overrides:
+      metrics_generator_processors:
+        - service-graphs
+        - span-metrics
+        - local-blocks
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: tempo
+  namespace: observability
+  labels:
+    app: tempo
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: tempo
+  template:
+    metadata:
+      labels:
+        app: tempo
+      annotations:
+        # Let Prometheus scrape Tempo's generated span/service-graph metrics.
+        prometheus.io/scrape: "true"
+        prometheus.io/port: "3200"
+        prometheus.io/path: /metrics
+    spec:
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 10001
+        fsGroup: 10001
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        - name: tempo
+          image: grafana/tempo:2.7.1
+          args:
+            - "-config.file=/etc/tempo/tempo.yaml"
+          ports:
+            - name: http
+              containerPort: 3200
+            - name: otlp-grpc
+              containerPort: 4317
+            - name: otlp-http
+              containerPort: 4318
+          securityContext:
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities:
+              drop:
+                - ALL
+          readinessProbe:
+            httpGet:
+              path: /ready
+              port: 3200
+            initialDelaySeconds: 15
+            periodSeconds: 10
+          volumeMounts:
+            - name: config
+              mountPath: /etc/tempo
+            - name: data
+              mountPath: /var/tempo
+      volumes:
+        - name: config
+          configMap:
+            name: tempo-config
+        - name: data
+          emptyDir: {}
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: tempo
+  namespace: observability
+  labels:
+    app: tempo
+spec:
+  selector:
+    app: tempo
+  type: ClusterIP
+  ports:
+    - name: http
+      port: 3200
+      targetPort: 3200
+    - name: otlp-grpc
+      port: 4317
+      targetPort: 4317
+    - name: otlp-http
+      port: 4318
+      targetPort: 4318

documentdb-playground/telemetry/local/scripts/deploy.sh

@@ -74,6 +74,21 @@ else
     )
   fi
 
+  # Tracing (spec.monitoring.tracing) needs a gateway image built with OTLP
+  # trace support (the feature landed in gateway 0.115). The chart derives the
+  # gateway tag from documentDbVersion. Override it here to pull a
+  # tracing-capable gateway; leave unset to keep the chart default (metrics
+  # still work, but no spans are produced by a pre-0.115 gateway).
+  DOCUMENTDB_VERSION="${DOCUMENTDB_VERSION:-}"
+  if [[ -n "$DOCUMENTDB_VERSION" ]]; then
+    echo "  Overriding gateway/extension version -> ${DOCUMENTDB_VERSION}"
+    HELM_IMAGE_FLAGS+=( --set "documentDbVersion=${DOCUMENTDB_VERSION}" )
+  else
+    echo "  NOTE: using chart-default gateway version. Tracing spans require a"
+    echo "        gateway >= 0.115 — set DOCUMENTDB_VERSION to a tracing-capable"
+    echo "        version if no traces appear in Tempo/Grafana."
+  fi
+
   echo "  Installing DocumentDB operator from ${OPERATOR_CHART_DIR}..."
   helm install documentdb-operator "$OPERATOR_CHART_DIR" \
     --namespace documentdb-operator \
@@ -124,4 +139,5 @@ echo ""
 echo "=== Deployment Complete ==="
 echo "Grafana:    kubectl port-forward svc/grafana 3000:3000 -n observability --context $CONTEXT"
 echo "Prometheus: kubectl port-forward svc/prometheus 9090:9090 -n observability --context $CONTEXT"
+echo "Tempo:      kubectl port-forward svc/tempo 3200:3200 -n observability --context $CONTEXT"
 echo "Validate:   ./scripts/validate.sh"