diff --git a/CHANGELOG.md b/CHANGELOG.md index e08dd1a49..46bdd3766 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,7 @@ ### Major Features - **io_uring opt-in feature gate**: A new `IOUring` feature gate (`spec.featureGates.IOUring: true`) enables PostgreSQL 18 asynchronous I/O (`io_method=io_uring`). Because the container runtime's default seccomp profile strips the `io_uring_setup/enter/register` syscalls, the operator also relaxes the postgres container seccomp profile — pointing the pods at a hardened Localhost profile that re-allows only those three syscalls — when the gate is on, so no external Kyverno policy is needed. It is **opt-in** (default off) since io_uring relaxes the sandbox. The Localhost profile path is operator-level config via the Helm value `operator.ioUring.seccompProfile` (default `profiles/documentdb-iouring.json`). See [io_uring documentation](docs/operator-public-documentation/io-uring.md) and the [feature playground](documentdb-playground/io-uring-feature/). - **Gateway OTLP metrics in the per-pod sidecar**: when `spec.monitoring.enabled=true`, the OTel Collector sidecar now exposes an OTLP/gRPC receiver on `127.0.0.1:4317` and the documentdb-gateway is configured (via `OTEL_EXPORTER_OTLP_ENDPOINT` and `OTEL_METRICS_ENABLED`) to push its `db_client_*` metrics there. The sidecar's existing prometheus exporter re-exports them alongside the existing `documentdb.postgres.up` sqlquery output, with per-pod attribution added by the collector's resource processor. No new CRD fields; this turns on automatically wherever monitoring was already enabled. +- **Gateway distributed tracing (opt-in)**: a new `spec.monitoring.tracing` stanza (`enabled`, `otlpEndpoint`, `sqlCommenterEnabled`) turns on the gateway's OTLP trace exporter (`OTEL_TRACES_ENABLED`) and adds a `traces` pipeline to the per-pod OTel Collector sidecar that pushes spans (OTLP/gRPC) to a central trace backend such as Tempo. Unlike metrics — which Prometheus scrapes from each sidecar — traces must be pushed, so `otlpEndpoint` is required. With `sqlCommenterEnabled`, the gateway injects a W3C `traceparent` SQL comment into its Postgres queries for trace/log correlation. Requires a gateway image built with trace support (gateway ≥ 0.115). - **Two-Phase Extension Upgrade**: New `spec.schemaVersion` field separates binary upgrades (`spec.documentDBVersion`) from irreversible schema migrations (`ALTER EXTENSION UPDATE`). The default behavior gives you a rollback-safe window — update the binary first, validate, then finalize the schema. Set `schemaVersion: "auto"` for single-step upgrades in development environments. See the [upgrade guide](docs/operator-public-documentation/preview/operations/upgrades.md) for details. ### Breaking Changes @@ -18,6 +19,7 @@ ### Playground & Examples - **io_uring feature playground**: New `documentdb-playground/io-uring-feature/` demonstrates the operator-native `IOUring` opt-in, modeled on the upstream cnpg-playground seccomp approach — a kind cluster that `extraMount`s the curated Localhost seccomp profile, a DaemonSet installer for real clusters, the DocumentDB CR with `spec.featureGates.IOUring: true`, and verification steps. No Kyverno policy is required. - **Container metrics reference collector**: The telemetry playground now includes a reference OpenTelemetry Collector DaemonSet under `documentdb-playground/telemetry/container-metrics/` for clusters that do not already collect kubelet-backed container metrics. It scrapes each node's local kubelet for container, pod, and node CPU/memory/network/filesystem metrics and exposes them via Prometheus. The production operator chart does not install this platform-level collector; tenant DocumentDB clusters do not receive kubelet privileges. +- **Tracing in the local telemetry playground**: `documentdb-playground/telemetry/local/` now deploys a Tempo trace backend in the `observability` namespace, enables `spec.monitoring.tracing` on the demo cluster, and provisions a Grafana Tempo datasource with trace→metrics correlation. `validate.sh` checks Tempo readiness and gateway spans; `deploy.sh` accepts `DOCUMENTDB_VERSION` to pull a tracing-capable gateway image. ### Testing infrastructure - **Unified E2E test suite ([#346](https://github.com/documentdb/documentdb-kubernetes-operator/pull/346))**: The four legacy end-to-end workflows (`test-integration.yml`, `test-E2E.yml`, `test-backup-and-restore.yml`, `test-upgrade-and-rollback.yml`) and their bash / JavaScript (mongosh) / Python (pymongo) glue have been replaced by a single Go / Ginkgo v2 / Gomega suite under `test/e2e/`. Specs are organised by CRD operation (lifecycle, scale, data, performance, backup, tls, feature gates, exposure, status, upgrade), reuse CloudNative-PG's `tests/utils` packages as a library, and speak the Mongo wire protocol via `go.mongodb.org/mongo-driver/v2`. diff --git a/documentdb-playground/telemetry/local/README.md b/documentdb-playground/telemetry/local/README.md index 695a92cc1..b7eb8f3f7 100644 --- a/documentdb-playground/telemetry/local/README.md +++ b/documentdb-playground/telemetry/local/README.md @@ -1,6 +1,6 @@ # DocumentDB Telemetry Playground (Local) -A metrics-focused observability stack for DocumentDB on a local Kind cluster. Deploys a 3-instance HA cluster with the in-pod OTel sidecar enabled, a reference container-metrics DaemonSet, and a pre-configured Grafana dashboard for container/pod resource metrics. +An observability stack for DocumentDB on a local Kind cluster. Deploys a 3-instance HA cluster with the in-pod OTel sidecar enabled, a reference container-metrics DaemonSet, **distributed tracing** (gateway spans → Tempo), and pre-configured Grafana dashboards + datasources for metrics and traces. ## Prerequisites @@ -26,6 +26,10 @@ kubectl port-forward svc/grafana 3000:3000 -n observability --context kind-docum kubectl port-forward svc/prometheus 9090:9090 -n observability --context kind-documentdb-telemetry # → http://localhost:9090 +# 3b. Open Tempo directly (optional — Grafana already queries it) +kubectl port-forward svc/tempo 3200:3200 -n observability --context kind-documentdb-telemetry +# → http://localhost:3200 + # 4. Validate data is flowing ./scripts/validate.sh ``` @@ -42,13 +46,14 @@ The operator chart is installed **from this branch** (`operator/documentdb-helm- | cert-manager | `cert-manager` | TLS certificate management | | DocumentDB operator | `documentdb-operator` | Operator + CNPG (Helm chart from this branch) | | DocumentDB HA cluster | `documentdb-preview-ns` | 1 primary + 2 streaming replicas | -| OTel Collector sidecar | `documentdb-preview-ns` | One per pod, injected by the operator's CNPG sidecar plugin when `spec.monitoring.enabled=true`. Receives gateway OTLP metrics and runs the `sqlquery` Postgres-health receiver. | +| OTel Collector sidecar | `documentdb-preview-ns` | One per pod, injected by the operator's CNPG sidecar plugin when `spec.monitoring.enabled=true`. Receives gateway OTLP metrics **and traces** and runs the `sqlquery` Postgres-health receiver. | | Container-metrics DaemonSet | `documentdb-operator` | One OTel Collector per node from `documentdb-playground/telemetry/container-metrics/`. Scrapes each node's local kubelet for container/pod/node CPU, memory, network, filesystem metrics. | | Prometheus | `observability` | Metrics storage + alerting rules; scrapes the per-pod sidecar and reference DaemonSet via annotation discovery | -| Grafana | `observability` | Dashboard (Internals — container/pod resources) | +| Tempo | `observability` | Trace storage. Receives OTLP spans pushed by the per-pod sidecars' traces pipeline and serves them to Grafana. Enabled by `spec.monitoring.tracing`. | +| Grafana | `observability` | Dashboards + Prometheus/Tempo datasources with trace→metrics correlation | | Traffic generators | `documentdb-preview-ns` | Read/write workload via mongosh | -There is **no central OTel Collector Deployment**. Per-pod sidecars handle pod-local signals (Postgres health, gateway OTLP); the playground's reference DaemonSet handles node-local kubelet scraping for container resource metrics. +There is **no central OTel Collector Deployment**. Per-pod sidecars handle pod-local signals (Postgres health, gateway OTLP metrics/traces); the playground's reference DaemonSet handles node-local kubelet scraping for container resource metrics. Metrics are *scraped* by Prometheus from each sidecar; traces are *pushed* (OTLP) from each sidecar to the central Tempo backend, because traces cannot be scraped. ## Architecture @@ -57,15 +62,17 @@ graph TB subgraph cluster["Kind Cluster (documentdb-telemetry)"] subgraph obs["observability namespace"] prometheus["Prometheus
annotation discovery"] - grafana["Grafana
:3000
Internals dashboard"] + tempo["Tempo
:3200 (query) / :4317 (OTLP)
traces store"] + grafana["Grafana
:3000
Prometheus + Tempo"] prometheus --> grafana + tempo --> grafana end subgraph docdb["documentdb-preview-ns"] subgraph pod1["Pod: preview-1 (primary)"] pg1["postgres :5432"] gw1["documentdb-gateway :10260"] - otel1["otel-collector sidecar
sqlquery + otlp / Prom :9188"] + otel1["otel-collector sidecar
sqlquery + otlp / Prom :9188
traces pipeline"] end subgraph pod2["Pod: preview-2 (replica)"] pg2["postgres :5432"] @@ -92,6 +99,9 @@ graph TB prometheus -- "scrape :9188 (annotation)" --> otel2 prometheus -- "scrape :9188 (annotation)" --> otel3 prometheus -- "scrape :8889 (annotation)" --> ds + otel1 -- "push spans OTLP :4317" --> tempo + otel2 -- "push spans OTLP :4317" --> tempo + otel3 -- "push spans OTLP :4317" --> tempo end user["Browser"] --> grafana @@ -107,8 +117,8 @@ local/ │ ├── validate.sh # Health check — verifies sidecar + data flow │ └── teardown.sh # Deletes cluster and proxy containers ├── k8s/ -│ ├── observability/ # Namespace, Prometheus (annotation discovery), Grafana -│ ├── documentdb/ # DocumentDB CR (with spec.monitoring.enabled) + credentials +│ ├── observability/ # Namespace, Prometheus (annotation discovery), Grafana, Tempo (traces) +│ ├── documentdb/ # DocumentDB CR (spec.monitoring.enabled + spec.monitoring.tracing) + credentials │ └── traffic/ # Traffic generator services + jobs └── dashboards/ └── internals.json # Container & pod resources dashboard (DaemonSet kubeletstats) @@ -126,6 +136,54 @@ One dashboard is auto-provisioned in the **DocumentDB** folder: Dashboards auto-refresh every 30 seconds. Edits made in the Grafana UI persist until the pod restarts. +## Tracing + +Distributed tracing is enabled via `spec.monitoring.tracing` on the DocumentDB CR +(`k8s/documentdb/cluster.yaml`): + +```yaml +monitoring: + enabled: true + exporter: + prometheus: + port: 9188 + tracing: + enabled: true + otlpEndpoint: tempo.observability.svc.cluster.local:4317 + sqlCommenterEnabled: true +``` + +When enabled, the operator: + +1. Sets `OTEL_TRACES_ENABLED=true` (and, with `sqlCommenterEnabled`, + `DOCUMENTDB_SQL_COMMENTER_ENABLED=true`) on the gateway container so it exports + OTLP spans to the co-located sidecar. +2. Adds a **traces pipeline** to the sidecar collector (`otlp` receiver → + `resource`/`batch` → `otlp/traces` exporter) that pushes spans to **Tempo**. + +Unlike metrics — which Prometheus *scrapes* from each sidecar — traces are +*pushed* to the central Tempo backend, since traces cannot be scraped. + +**View traces:** Grafana → **Explore → Tempo**, search `service.name = documentdb_gateway`. +A single request produces `gateway.request → gateway.process_request → +postgres.*` spans. With `sqlCommenterEnabled`, gateway-issued Postgres queries +carry a `/* traceparent=… */` comment so database logs correlate to the trace. + +> **Gateway image requirement:** tracing requires a gateway built with OTLP +> trace support (the feature landed in gateway **0.115**). The chart's default +> gateway version predates it. Set `DOCUMENTDB_VERSION` to a tracing-capable +> version when deploying, e.g.: +> +> ```bash +> DOCUMENTDB_VERSION=0.115.0 ./scripts/deploy.sh +> ``` +> +> With a pre-0.115 gateway the pipeline is wired correctly but no spans are +> produced (`validate.sh` reports this as a warning, not a failure). For +> client → gateway → Postgres trace linking, use a fully OTel-instrumented +> client that passes a W3C `traceparent` in the Mongo `comment` field (the +> mongosh-based traffic generators here do not emit client spans). + ## Alerting Rules Prometheus includes a sample alerting rule: diff --git a/documentdb-playground/telemetry/local/k8s/documentdb/cluster.yaml b/documentdb-playground/telemetry/local/k8s/documentdb/cluster.yaml index 082051177..99d4d9ed7 100644 --- a/documentdb-playground/telemetry/local/k8s/documentdb/cluster.yaml +++ b/documentdb-playground/telemetry/local/k8s/documentdb/cluster.yaml @@ -29,7 +29,8 @@ spec: resource: storage: pvcSize: 5Gi - sidecarInjectorPluginName: cnpg-i-sidecar-injector.documentdb.io + plugins: + sidecarInjectorName: cnpg-i-sidecar-injector.documentdb.io # Enable the OTel Collector sidecar (one per pod). The sidecar: # - receives OTLP metrics from the documentdb-gateway on localhost:4317 # - exposes Prometheus /metrics on the configured port @@ -43,3 +44,17 @@ spec: exporter: prometheus: port: 9188 + # Distributed tracing (opt-in). The gateway exports OTLP spans to the + # co-located sidecar, which forwards them to the central Tempo backend in + # the observability namespace. Unlike metrics (scraped by Prometheus), + # traces must be pushed to a trace store. + # + # NOTE: requires a gateway image built with tracing support (>= 0.115). + # The operator's default gateway image predates this; override it via the + # chart value image.gateway (or spec.image.gateway) when deploying. + tracing: + enabled: true + otlpEndpoint: tempo.observability.svc.cluster.local:4317 + # Inject a W3C traceparent SQL comment into gateway-issued Postgres + # queries so PostgreSQL logs correlate to gateway traces. + sqlCommenterEnabled: true diff --git a/documentdb-playground/telemetry/local/k8s/observability/grafana.yaml b/documentdb-playground/telemetry/local/k8s/observability/grafana.yaml index 5fd5cc237..3c7dce671 100644 --- a/documentdb-playground/telemetry/local/k8s/observability/grafana.yaml +++ b/documentdb-playground/telemetry/local/k8s/observability/grafana.yaml @@ -65,6 +65,19 @@ data: access: proxy url: http://prometheus.observability.svc:9090 isDefault: true + - name: Tempo + type: tempo + uid: tempo + access: proxy + url: http://tempo.observability.svc:3200 + jsonData: + # Trace -> metrics correlation via Tempo-generated span metrics. + tracesToMetrics: + datasourceUid: prometheus + serviceMap: + datasourceUid: prometheus + nodeGraph: + enabled: true --- apiVersion: v1 kind: ConfigMap diff --git a/documentdb-playground/telemetry/local/k8s/observability/tempo.yaml b/documentdb-playground/telemetry/local/k8s/observability/tempo.yaml new file mode 100644 index 000000000..e23c3ad69 --- /dev/null +++ b/documentdb-playground/telemetry/local/k8s/observability/tempo.yaml @@ -0,0 +1,146 @@ +# ============================================================ +# Tempo - Distributed tracing backend (traces store) +# ⚠️ DEMO/PLAYGROUND ONLY — single-binary, local ephemeral +# storage (emptyDir), no auth. Not for production use. +# ============================================================ +# +# Why Tempo (and not Prometheus) for traces: +# Metrics are *scraped* by Prometheus from each per-pod OTel sidecar. +# Traces cannot be scraped — each sidecar *pushes* spans (OTLP/gRPC) to a +# central store. Tempo receives OTLP on :4317 and Grafana queries it on :3200. +# +# The DocumentDB CR points the sidecar traces pipeline at +# tempo.observability.svc.cluster.local:4317 +# via spec.monitoring.tracing.otlpEndpoint. +apiVersion: v1 +kind: ConfigMap +metadata: + name: tempo-config + namespace: observability +data: + tempo.yaml: | + server: + http_listen_port: 3200 + grpc_listen_port: 9096 + distributor: + receivers: + otlp: + protocols: + grpc: + endpoint: 0.0.0.0:4317 + http: + endpoint: 0.0.0.0:4318 + ingester: + max_block_duration: 5m + compactor: + compaction: + block_retention: 1h + storage: + trace: + backend: local + local: + path: /var/tempo/blocks + wal: + path: /var/tempo/wal + # Generate service-graph and span metrics so Grafana's trace->metrics and + # node-graph views work. Metrics are exposed for Prometheus to scrape. + metrics_generator: + registry: + external_labels: + source: tempo + storage: + path: /var/tempo/generator/wal + processor: + local_blocks: + filter_server_spans: false + overrides: + metrics_generator_processors: + - service-graphs + - span-metrics + - local-blocks +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: tempo + namespace: observability + labels: + app: tempo +spec: + replicas: 1 + selector: + matchLabels: + app: tempo + template: + metadata: + labels: + app: tempo + annotations: + # Let Prometheus scrape Tempo's generated span/service-graph metrics. + prometheus.io/scrape: "true" + prometheus.io/port: "3200" + prometheus.io/path: /metrics + spec: + securityContext: + runAsNonRoot: true + runAsUser: 10001 + fsGroup: 10001 + seccompProfile: + type: RuntimeDefault + containers: + - name: tempo + image: grafana/tempo:2.7.1 + args: + - "-config.file=/etc/tempo/tempo.yaml" + ports: + - name: http + containerPort: 3200 + - name: otlp-grpc + containerPort: 4317 + - name: otlp-http + containerPort: 4318 + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: + - ALL + readinessProbe: + httpGet: + path: /ready + port: 3200 + initialDelaySeconds: 15 + periodSeconds: 10 + volumeMounts: + - name: config + mountPath: /etc/tempo + - name: data + mountPath: /var/tempo + volumes: + - name: config + configMap: + name: tempo-config + - name: data + emptyDir: {} +--- +apiVersion: v1 +kind: Service +metadata: + name: tempo + namespace: observability + labels: + app: tempo +spec: + selector: + app: tempo + type: ClusterIP + ports: + - name: http + port: 3200 + targetPort: 3200 + - name: otlp-grpc + port: 4317 + targetPort: 4317 + - name: otlp-http + port: 4318 + targetPort: 4318 diff --git a/documentdb-playground/telemetry/local/scripts/deploy.sh b/documentdb-playground/telemetry/local/scripts/deploy.sh index d011a20c6..314df7b99 100755 --- a/documentdb-playground/telemetry/local/scripts/deploy.sh +++ b/documentdb-playground/telemetry/local/scripts/deploy.sh @@ -74,6 +74,21 @@ else ) fi + # Tracing (spec.monitoring.tracing) needs a gateway image built with OTLP + # trace support (the feature landed in gateway 0.115). The chart derives the + # gateway tag from documentDbVersion. Override it here to pull a + # tracing-capable gateway; leave unset to keep the chart default (metrics + # still work, but no spans are produced by a pre-0.115 gateway). + DOCUMENTDB_VERSION="${DOCUMENTDB_VERSION:-}" + if [[ -n "$DOCUMENTDB_VERSION" ]]; then + echo " Overriding gateway/extension version -> ${DOCUMENTDB_VERSION}" + HELM_IMAGE_FLAGS+=( --set "documentDbVersion=${DOCUMENTDB_VERSION}" ) + else + echo " NOTE: using chart-default gateway version. Tracing spans require a" + echo " gateway >= 0.115 — set DOCUMENTDB_VERSION to a tracing-capable" + echo " version if no traces appear in Tempo/Grafana." + fi + echo " Installing DocumentDB operator from ${OPERATOR_CHART_DIR}..." helm install documentdb-operator "$OPERATOR_CHART_DIR" \ --namespace documentdb-operator \ @@ -124,4 +139,5 @@ echo "" echo "=== Deployment Complete ===" echo "Grafana: kubectl port-forward svc/grafana 3000:3000 -n observability --context $CONTEXT" echo "Prometheus: kubectl port-forward svc/prometheus 9090:9090 -n observability --context $CONTEXT" +echo "Tempo: kubectl port-forward svc/tempo 3200:3200 -n observability --context $CONTEXT" echo "Validate: ./scripts/validate.sh" diff --git a/documentdb-playground/telemetry/local/scripts/validate.sh b/documentdb-playground/telemetry/local/scripts/validate.sh index 9562ff81d..8d3be7ee4 100755 --- a/documentdb-playground/telemetry/local/scripts/validate.sh +++ b/documentdb-playground/telemetry/local/scripts/validate.sh @@ -15,7 +15,7 @@ echo "" # 1. Check observability deployments (no central OTel collector — every # DocumentDB pod runs its own sidecar via spec.monitoring). echo "--- Observability Stack ---" -for deploy in prometheus grafana; do +for deploy in prometheus grafana tempo; do if kubectl get deployment "$deploy" -n observability --context "$CONTEXT" &>/dev/null; then ready=$(kubectl get deployment "$deploy" -n observability --context "$CONTEXT" -o jsonpath='{.status.readyReplicas}' 2>/dev/null || echo "0") if [ "${ready:-0}" -ge 1 ]; then @@ -107,6 +107,32 @@ else red "Prometheus pod not found" fi +# 4. Check tracing: Tempo readiness + gateway spans present. +echo "" +echo "--- Tracing ---" +TEMPO_POD=$(kubectl get pod -l app=tempo -n observability --context "$CONTEXT" -o jsonpath='{.items[0].metadata.name}' 2>/dev/null || echo "") +if [ -z "$TEMPO_POD" ]; then + warn "Tempo pod not found — tracing backend not deployed" +else + if kubectl exec "$TEMPO_POD" -n observability --context "$CONTEXT" -- \ + wget -qO- "http://localhost:3200/ready" 2>/dev/null | grep -q "ready"; then + green "Tempo is ready" + else + warn "Tempo not ready yet (may still be starting)" + fi + + # Query Tempo for any gateway-emitted spans. Absence is a warning (not a + # failure): a pre-0.115 gateway image produces no spans even though the + # pipeline is wired correctly (see DOCUMENTDB_VERSION in deploy.sh). + traces=$(kubectl exec "$TEMPO_POD" -n observability --context "$CONTEXT" -- \ + wget -qO- "http://localhost:3200/api/search?tags=service.name%3Ddocumentdb_gateway&limit=1" 2>/dev/null || echo "") + if echo "$traces" | grep -q '"traceID"'; then + green "Tempo has documentdb_gateway traces" + else + warn "No gateway traces in Tempo yet — send traffic, and ensure the gateway image supports tracing (DOCUMENTDB_VERSION >= 0.115)." + fi +fi + # Summary echo "" echo "=== Results: $PASS passed, $FAIL failed ===" diff --git a/documentdb-playground/telemetry/scripts/create-cluster.sh b/documentdb-playground/telemetry/scripts/create-cluster.sh index e6696ec45..d02d61be3 100755 --- a/documentdb-playground/telemetry/scripts/create-cluster.sh +++ b/documentdb-playground/telemetry/scripts/create-cluster.sh @@ -485,7 +485,8 @@ spec: storageClass: documentdb-storage # Custom Premium SSD exposeViaService: serviceType: LoadBalancer - sidecarInjectorPluginName: cnpg-i-sidecar-injector.documentdb.io + plugins: + sidecarInjectorName: cnpg-i-sidecar-injector.documentdb.io EOF else # Use AKS default storage (StandardSSD_LRS) @@ -506,7 +507,8 @@ spec: # storageClass omitted - uses AKS default (StandardSSD_LRS) exposeViaService: serviceType: LoadBalancer - sidecarInjectorPluginName: cnpg-i-sidecar-injector.documentdb.io + plugins: + sidecarInjectorName: cnpg-i-sidecar-injector.documentdb.io EOF fi diff --git a/operator/cnpg-plugins/sidecar-injector/internal/config/config.go b/operator/cnpg-plugins/sidecar-injector/internal/config/config.go index a2903fb17..e1970fc88 100644 --- a/operator/cnpg-plugins/sidecar-injector/internal/config/config.go +++ b/operator/cnpg-plugins/sidecar-injector/internal/config/config.go @@ -23,6 +23,8 @@ const ( otelCollectorImageParameter = "otelCollectorImage" otelConfigMapNameParameter = "otelConfigMapName" prometheusPortParameter = "prometheusPort" + gatewayTracingEnabledParameter = "gatewayTracingEnabled" + gatewaySQLCommenterEnabledParameter = "gatewaySqlCommenterEnabled" ) // Configuration represents the plugin configuration parameters @@ -35,6 +37,12 @@ type Configuration struct { OtelCollectorImage string OtelConfigMapName string PrometheusPort int32 + // GatewayTracingEnabled turns on the gateway's OTLP trace exporter + // (OTEL_TRACES_ENABLED) so it pushes spans to the sidecar collector. + GatewayTracingEnabled bool + // GatewaySQLCommenterEnabled injects a W3C traceparent SQL comment into + // gateway-issued Postgres queries for trace/log correlation. + GatewaySQLCommenterEnabled bool } // FromParameters builds a plugin configuration from the configuration parameters @@ -90,6 +98,8 @@ func FromParameters( OtelCollectorImage: helper.Parameters[otelCollectorImageParameter], OtelConfigMapName: helper.Parameters[otelConfigMapNameParameter], PrometheusPort: prometheusPort, + GatewayTracingEnabled: helper.Parameters[gatewayTracingEnabledParameter] == "true", + GatewaySQLCommenterEnabled: helper.Parameters[gatewaySQLCommenterEnabledParameter] == "true", } configuration.applyDefaults() diff --git a/operator/cnpg-plugins/sidecar-injector/internal/lifecycle/lifecycle.go b/operator/cnpg-plugins/sidecar-injector/internal/lifecycle/lifecycle.go index f028741cf..6afb1ce4f 100644 --- a/operator/cnpg-plugins/sidecar-injector/internal/lifecycle/lifecycle.go +++ b/operator/cnpg-plugins/sidecar-injector/internal/lifecycle/lifecycle.go @@ -296,11 +296,11 @@ func (impl Implementation) reconcileMetadata( } // Set OTel-related env vars on the gateway container so it can push its - // own metrics (traces pipeline TBD) to the co-located OTel Collector + // own metrics (and traces, when enabled) to the co-located OTel Collector // sidecar and so that every signal carries a per-pod service.instance.id // resource attribute. Only set when the sidecar is present to avoid // connection errors and resource-attribute drift. - injectGatewayOTelEnv(mutatedPod) + injectGatewayOTelEnv(mutatedPod, configuration) log.Printf("OTel Collector sidecar injected successfully") } @@ -366,14 +366,14 @@ func hardenedSecurityContext() *corev1.SecurityContext { } // gatewayOTelEnvVars returns the OTel-related env vars that the sidecar -// injector adds to the gateway container so it can push metrics to the -// co-located OTel Collector sidecar. +// injector adds to the gateway container so it can push metrics (and, when +// tracing is enabled, spans) to the co-located OTel Collector sidecar. // // Per-pod attribution (k8s.pod.name) is added by the collector's resource -// processor on every exported metric, so we don't need to set +// processor on every exported signal, so we don't need to set // OTEL_RESOURCE_ATTRIBUTES / service.instance.id here. -func gatewayOTelEnvVars() []corev1.EnvVar { - return []corev1.EnvVar{ +func gatewayOTelEnvVars(cfg *config.Configuration) []corev1.EnvVar { + envs := []corev1.EnvVar{ { Name: "OTEL_EXPORTER_OTLP_ENDPOINT", Value: "http://127.0.0.1:4317", @@ -386,6 +386,26 @@ func gatewayOTelEnvVars() []corev1.EnvVar { Value: "true", }, } + + // Tracing is opt-in (spec.monitoring.tracing.enabled). The gateway gates + // its OTLP trace exporter behind OTEL_TRACES_ENABLED; the matching traces + // pipeline is added to the sidecar collector by the operator. + if cfg != nil && cfg.GatewayTracingEnabled { + envs = append(envs, corev1.EnvVar{ + Name: "OTEL_TRACES_ENABLED", + Value: "true", + }) + if cfg.GatewaySQLCommenterEnabled { + // Injects a W3C traceparent SQL comment into gateway-issued + // Postgres queries so database logs correlate to traces. + envs = append(envs, corev1.EnvVar{ + Name: "DOCUMENTDB_SQL_COMMENTER_ENABLED", + Value: "true", + }) + } + } + + return envs } // injectGatewayOTelEnv mutates `pod` to append OTel env vars to the gateway @@ -397,8 +417,8 @@ func gatewayOTelEnvVars() []corev1.EnvVar { // entries and CNPG's pod metadata reconciler would fail with // "Pod is invalid: spec: Forbidden: pod updates may not change fields other // than ...". -func injectGatewayOTelEnv(pod *corev1.Pod) { - envs := gatewayOTelEnvVars() +func injectGatewayOTelEnv(pod *corev1.Pod, cfg *config.Configuration) { + envs := gatewayOTelEnvVars(cfg) for i := range pod.Spec.Containers { if pod.Spec.Containers[i].Name != gatewayContainerName { continue diff --git a/operator/cnpg-plugins/sidecar-injector/internal/lifecycle/lifecycle_test.go b/operator/cnpg-plugins/sidecar-injector/internal/lifecycle/lifecycle_test.go index 044edf42c..5449386f3 100644 --- a/operator/cnpg-plugins/sidecar-injector/internal/lifecycle/lifecycle_test.go +++ b/operator/cnpg-plugins/sidecar-injector/internal/lifecycle/lifecycle_test.go @@ -7,6 +7,7 @@ import ( "reflect" "testing" + "github.com/documentdb/cnpg-i-sidecar-injector/internal/config" corev1 "k8s.io/api/core/v1" ) @@ -36,7 +37,7 @@ func envNames(env []corev1.EnvVar) []string { // so we don't need POD_NAME / OTEL_RESOURCE_ATTRIBUTES on the gateway. func TestInjectGatewayOTelEnv_AllAppended(t *testing.T) { pod := gatewayContainer() - injectGatewayOTelEnv(pod) + injectGatewayOTelEnv(pod, nil) got := envNames(pod.Spec.Containers[1].Env) want := []string{"OTEL_EXPORTER_OTLP_ENDPOINT", "OTEL_METRICS_ENABLED"} @@ -50,7 +51,7 @@ func TestInjectGatewayOTelEnv_AllAppended(t *testing.T) { // silently disables OTLP metrics export. func TestInjectGatewayOTelEnv_MetricsEnabledPresent(t *testing.T) { pod := gatewayContainer() - injectGatewayOTelEnv(pod) + injectGatewayOTelEnv(pod, nil) for _, e := range pod.Spec.Containers[1].Env { if e.Name == "OTEL_METRICS_ENABLED" { if e.Value != "true" { @@ -67,7 +68,7 @@ func TestInjectGatewayOTelEnv_MetricsEnabledPresent(t *testing.T) { // untouched (we don't overwrite), and the others are still appended. func TestInjectGatewayOTelEnv_PreservesExisting(t *testing.T) { pod := gatewayContainer(corev1.EnvVar{Name: "OTEL_EXPORTER_OTLP_ENDPOINT", Value: "http://custom:4317"}) - injectGatewayOTelEnv(pod) + injectGatewayOTelEnv(pod, nil) env := pod.Spec.Containers[1].Env if len(env) != 2 { @@ -87,10 +88,10 @@ func TestInjectGatewayOTelEnv_PreservesExisting(t *testing.T) { // may not change fields other than ...". func TestInjectGatewayOTelEnv_Idempotent(t *testing.T) { pod := gatewayContainer() - injectGatewayOTelEnv(pod) + injectGatewayOTelEnv(pod, nil) first := append([]corev1.EnvVar(nil), pod.Spec.Containers[1].Env...) - injectGatewayOTelEnv(pod) + injectGatewayOTelEnv(pod, nil) second := pod.Spec.Containers[1].Env if !reflect.DeepEqual(first, second) { @@ -105,12 +106,73 @@ func TestInjectGatewayOTelEnv_NoGatewayContainer(t *testing.T) { Containers: []corev1.Container{{Name: "postgres"}}, }, } - injectGatewayOTelEnv(pod) + injectGatewayOTelEnv(pod, nil) if len(pod.Spec.Containers[0].Env) != 0 { t.Errorf("expected no envs on non-gateway container, got %v", envNames(pod.Spec.Containers[0].Env)) } } +// TestInjectGatewayOTelEnv_TracingDisabledByDefault verifies that a config +// without tracing yields only the metrics env vars (no OTEL_TRACES_ENABLED). +func TestInjectGatewayOTelEnv_TracingDisabledByDefault(t *testing.T) { + pod := gatewayContainer() + injectGatewayOTelEnv(pod, &config.Configuration{}) + + got := envNames(pod.Spec.Containers[1].Env) + want := []string{"OTEL_EXPORTER_OTLP_ENDPOINT", "OTEL_METRICS_ENABLED"} + if !reflect.DeepEqual(got, want) { + t.Errorf("envs got %v, want %v", got, want) + } +} + +// TestInjectGatewayOTelEnv_TracingEnabled verifies that enabling tracing adds +// OTEL_TRACES_ENABLED (the pgmongo gateway gates its trace exporter on it). +func TestInjectGatewayOTelEnv_TracingEnabled(t *testing.T) { + pod := gatewayContainer() + injectGatewayOTelEnv(pod, &config.Configuration{GatewayTracingEnabled: true}) + + got := envNames(pod.Spec.Containers[1].Env) + want := []string{"OTEL_EXPORTER_OTLP_ENDPOINT", "OTEL_METRICS_ENABLED", "OTEL_TRACES_ENABLED"} + if !reflect.DeepEqual(got, want) { + t.Errorf("envs got %v, want %v", got, want) + } +} + +// TestInjectGatewayOTelEnv_SQLCommenterEnabled verifies the optional +// DOCUMENTDB_SQL_COMMENTER_ENABLED env is added only when both tracing and +// SQLCommenter are on. +func TestInjectGatewayOTelEnv_SQLCommenterEnabled(t *testing.T) { + pod := gatewayContainer() + injectGatewayOTelEnv(pod, &config.Configuration{ + GatewayTracingEnabled: true, + GatewaySQLCommenterEnabled: true, + }) + + got := envNames(pod.Spec.Containers[1].Env) + want := []string{ + "OTEL_EXPORTER_OTLP_ENDPOINT", + "OTEL_METRICS_ENABLED", + "OTEL_TRACES_ENABLED", + "DOCUMENTDB_SQL_COMMENTER_ENABLED", + } + if !reflect.DeepEqual(got, want) { + t.Errorf("envs got %v, want %v", got, want) + } +} + +// TestInjectGatewayOTelEnv_SQLCommenterRequiresTracing verifies SQLCommenter +// is not injected when tracing itself is disabled (traceparent needs a span). +func TestInjectGatewayOTelEnv_SQLCommenterRequiresTracing(t *testing.T) { + pod := gatewayContainer() + injectGatewayOTelEnv(pod, &config.Configuration{GatewaySQLCommenterEnabled: true}) + + for _, e := range pod.Spec.Containers[1].Env { + if e.Name == "DOCUMENTDB_SQL_COMMENTER_ENABLED" { + t.Error("DOCUMENTDB_SQL_COMMENTER_ENABLED must not be set when tracing is disabled") + } + } +} + // assertPSARestricted verifies a container SecurityContext sets every field // required by the Kubernetes Pod Security Admission "restricted" profile. // Pod-level inheritance does not satisfy PSA, so the injected sidecars must diff --git a/operator/documentdb-helm-chart/crds/documentdb.io_dbs.yaml b/operator/documentdb-helm-chart/crds/documentdb.io_dbs.yaml index 672830030..2e7c357dc 100644 --- a/operator/documentdb-helm-chart/crds/documentdb.io_dbs.yaml +++ b/operator/documentdb-helm-chart/crds/documentdb.io_dbs.yaml @@ -1286,6 +1286,26 @@ spec: type: integer type: object type: object + tracing: + description: |- + Tracing configures distributed tracing (spans) export from the gateway. + Requires Enabled=true (the traces share the same sidecar collector). + properties: + enabled: + description: Enabled turns on gateway trace export and the + sidecar traces pipeline. + type: boolean + otlpEndpoint: + description: |- + OTLPEndpoint is the OTLP/gRPC endpoint the sidecar pushes spans to + (e.g., "tempo.observability:4317"). Required when Enabled is true. + type: string + sqlCommenterEnabled: + description: |- + SQLCommenterEnabled injects a W3C `traceparent` SQL comment into + gateway-issued Postgres queries so database logs correlate to traces. + type: boolean + type: object type: object nodeCount: description: NodeCount is the number of nodes in the DocumentDB cluster. diff --git a/operator/src/api/preview/documentdb_types.go b/operator/src/api/preview/documentdb_types.go index b0cae72f1..094a2b603 100644 --- a/operator/src/api/preview/documentdb_types.go +++ b/operator/src/api/preview/documentdb_types.go @@ -416,6 +416,31 @@ type MonitoringSpec struct { // Exporter configures where metrics are sent. // +optional Exporter *ExporterSpec `json:"exporter,omitempty"` + + // Tracing configures distributed tracing (spans) export from the gateway. + // Requires Enabled=true (the traces share the same sidecar collector). + // +optional + Tracing *TracingSpec `json:"tracing,omitempty"` +} + +// TracingSpec configures distributed tracing from the documentdb-gateway. When +// enabled, the operator turns on the gateway's OTLP trace exporter and adds a +// traces pipeline to the OTel Collector sidecar that pushes spans (OTLP/gRPC) +// to OTLPEndpoint. Unlike metrics — which Prometheus scrapes from each sidecar — +// traces must be pushed to a central trace backend (e.g. Tempo or Jaeger). +type TracingSpec struct { + // Enabled turns on gateway trace export and the sidecar traces pipeline. + Enabled bool `json:"enabled,omitempty"` + + // OTLPEndpoint is the OTLP/gRPC endpoint the sidecar pushes spans to + // (e.g., "tempo.observability:4317"). Required when Enabled is true. + // +optional + OTLPEndpoint string `json:"otlpEndpoint,omitempty"` + + // SQLCommenterEnabled injects a W3C `traceparent` SQL comment into + // gateway-issued Postgres queries so database logs correlate to traces. + // +optional + SQLCommenterEnabled bool `json:"sqlCommenterEnabled,omitempty"` } // ExporterSpec configures metric export destinations. diff --git a/operator/src/api/preview/zz_generated.deepcopy.go b/operator/src/api/preview/zz_generated.deepcopy.go index 804104c90..a56d81eca 100644 --- a/operator/src/api/preview/zz_generated.deepcopy.go +++ b/operator/src/api/preview/zz_generated.deepcopy.go @@ -478,6 +478,11 @@ func (in *MonitoringSpec) DeepCopyInto(out *MonitoringSpec) { *out = new(ExporterSpec) (*in).DeepCopyInto(*out) } + if in.Tracing != nil { + in, out := &in.Tracing, &out.Tracing + *out = new(TracingSpec) + **out = **in + } } // DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new MonitoringSpec. @@ -831,3 +836,18 @@ func (in *Timeouts) DeepCopy() *Timeouts { in.DeepCopyInto(out) return out } + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *TracingSpec) DeepCopyInto(out *TracingSpec) { + *out = *in +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new TracingSpec. +func (in *TracingSpec) DeepCopy() *TracingSpec { + if in == nil { + return nil + } + out := new(TracingSpec) + in.DeepCopyInto(out) + return out +} diff --git a/operator/src/config/crd/bases/documentdb.io_dbs.yaml b/operator/src/config/crd/bases/documentdb.io_dbs.yaml index 672830030..2e7c357dc 100644 --- a/operator/src/config/crd/bases/documentdb.io_dbs.yaml +++ b/operator/src/config/crd/bases/documentdb.io_dbs.yaml @@ -1286,6 +1286,26 @@ spec: type: integer type: object type: object + tracing: + description: |- + Tracing configures distributed tracing (spans) export from the gateway. + Requires Enabled=true (the traces share the same sidecar collector). + properties: + enabled: + description: Enabled turns on gateway trace export and the + sidecar traces pipeline. + type: boolean + otlpEndpoint: + description: |- + OTLPEndpoint is the OTLP/gRPC endpoint the sidecar pushes spans to + (e.g., "tempo.observability:4317"). Required when Enabled is true. + type: string + sqlCommenterEnabled: + description: |- + SQLCommenterEnabled injects a W3C `traceparent` SQL comment into + gateway-issued Postgres queries so database logs correlate to traces. + type: boolean + type: object type: object nodeCount: description: NodeCount is the number of nodes in the DocumentDB cluster. diff --git a/operator/src/internal/cnpg/cnpg_cluster.go b/operator/src/internal/cnpg/cnpg_cluster.go index 591c5410b..e60ee8a19 100644 --- a/operator/src/internal/cnpg/cnpg_cluster.go +++ b/operator/src/internal/cnpg/cnpg_cluster.go @@ -106,6 +106,15 @@ func GetCnpgClusterSpec(req ctrl.Request, documentdb *dbpreview.DocumentDB, docu } else { log.Error(err, "Failed to generate OTel config hash; config changes may not trigger rolling restart") } + // Tracing is opt-in and shares the sidecar collector. Surface + // it as plugin parameters so the injector enables the gateway's + // OTLP trace exporter (and optional SQLCommenter) env vars. + if tracing := documentdb.Spec.Monitoring.Tracing; tracing != nil && tracing.Enabled { + params["gatewayTracingEnabled"] = "true" + if tracing.SQLCommenterEnabled { + params["gatewaySqlCommenterEnabled"] = "true" + } + } } return []cnpgv1.PluginConfiguration{{ Name: sidecarPluginName, diff --git a/operator/src/internal/otel/config.go b/operator/src/internal/otel/config.go index 59c1584d7..ebc53de47 100644 --- a/operator/src/internal/otel/config.go +++ b/operator/src/internal/otel/config.go @@ -139,23 +139,53 @@ func generateDynamicConfig(clusterName, namespace string, spec *dbpreview.Monito } } - // Wire pipeline: receivers + batch from static.yaml, resource from this dynamic config. - // Disable the collector's internal telemetry to avoid port conflicts with the - // Prometheus exporter (both default to 8888). - if len(exporterNames) > 0 { + // Traces pipeline: the gateway pushes spans to the sidecar's otlp receiver + // and the sidecar forwards them (OTLP/gRPC) to a central trace backend. + // Unlike metrics, traces cannot be scraped by Prometheus and must be + // exported to a store such as Tempo or Jaeger. + tracesEnabled := false + if spec.Tracing != nil && spec.Tracing.Enabled && spec.Tracing.OTLPEndpoint != "" { + if cfg.Exporters == nil { + cfg.Exporters = map[string]any{} + } + cfg.Exporters["otlp/traces"] = map[string]any{ + "endpoint": spec.Tracing.OTLPEndpoint, + "tls": map[string]any{ + // TODO: Support TLS for the traces exporter alongside the + // metrics OTLP exporter. Insecure is fine for in-cluster paths. + "insecure": true, + }, + } + tracesEnabled = true + } + + // Wire pipelines: receivers + batch from static.yaml, resource from this + // dynamic config. Disable the collector's internal telemetry to avoid port + // conflicts with the Prometheus exporter (both default to 8888). + if len(exporterNames) > 0 || tracesEnabled { + pipelines := map[string]pipelineConfig{} + if len(exporterNames) > 0 { + pipelines["metrics"] = pipelineConfig{ + Receivers: receiverNames, + Processors: []string{"resource", "batch"}, + Exporters: exporterNames, + } + } + if tracesEnabled { + // Only the otlp receiver carries gateway spans; sqlquery is metrics-only. + pipelines["traces"] = pipelineConfig{ + Receivers: []string{"otlp"}, + Processors: []string{"resource", "batch"}, + Exporters: []string{"otlp/traces"}, + } + } cfg.Service = &serviceConfig{ Telemetry: &telemetryConfig{ Metrics: &telemetryMetricsConfig{ Level: "none", }, }, - Pipelines: map[string]pipelineConfig{ - "metrics": { - Receivers: receiverNames, - Processors: []string{"resource", "batch"}, - Exporters: exporterNames, - }, - }, + Pipelines: pipelines, } } diff --git a/operator/src/internal/otel/config_test.go b/operator/src/internal/otel/config_test.go index e44502650..a553c76ad 100644 --- a/operator/src/internal/otel/config_test.go +++ b/operator/src/internal/otel/config_test.go @@ -196,6 +196,67 @@ var _ = Describe("GenerateConfigMapData", func() { Expect(dynCfg.Exporters).To(BeEmpty()) }) + It("adds a traces pipeline and otlp/traces exporter when tracing is enabled", func() { + spec := &dbpreview.MonitoringSpec{ + Enabled: true, + Exporter: &dbpreview.ExporterSpec{ + Prometheus: &dbpreview.PrometheusExporterSpec{Port: 9090}, + }, + Tracing: &dbpreview.TracingSpec{ + Enabled: true, + OTLPEndpoint: "tempo.observability:4317", + }, + } + data, err := GenerateConfigMapData("cluster", "ns", spec) + Expect(err).NotTo(HaveOccurred()) + + dynCfg := parseCfg(data["dynamic.yaml"]) + Expect(dynCfg.Exporters).To(HaveKey("otlp/traces")) + traceExp, ok := dynCfg.Exporters["otlp/traces"].(map[string]any) + Expect(ok).To(BeTrue()) + Expect(traceExp["endpoint"]).To(Equal("tempo.observability:4317")) + + Expect(dynCfg.Service.Pipelines).To(HaveKey("traces")) + Expect(dynCfg.Service.Pipelines["traces"].Exporters).To(ContainElement("otlp/traces")) + // sqlquery is metrics-only; the traces pipeline must use only the otlp receiver. + Expect(dynCfg.Service.Pipelines["traces"].Receivers).To(Equal([]string{"otlp"})) + // The metrics pipeline still exists alongside traces. + Expect(dynCfg.Service.Pipelines).To(HaveKey("metrics")) + }) + + It("skips the traces pipeline when tracing is enabled but endpoint is empty", func() { + spec := &dbpreview.MonitoringSpec{ + Enabled: true, + Exporter: &dbpreview.ExporterSpec{ + Prometheus: &dbpreview.PrometheusExporterSpec{Port: 9090}, + }, + Tracing: &dbpreview.TracingSpec{Enabled: true, OTLPEndpoint: ""}, + } + data, err := GenerateConfigMapData("cluster", "ns", spec) + Expect(err).NotTo(HaveOccurred()) + + dynCfg := parseCfg(data["dynamic.yaml"]) + Expect(dynCfg.Exporters).NotTo(HaveKey("otlp/traces")) + Expect(dynCfg.Service.Pipelines).NotTo(HaveKey("traces")) + }) + + It("generates a traces pipeline even when no metrics exporters are configured", func() { + spec := &dbpreview.MonitoringSpec{ + Enabled: true, + Tracing: &dbpreview.TracingSpec{ + Enabled: true, + OTLPEndpoint: "tempo.observability:4317", + }, + } + data, err := GenerateConfigMapData("cluster", "ns", spec) + Expect(err).NotTo(HaveOccurred()) + + dynCfg := parseCfg(data["dynamic.yaml"]) + Expect(dynCfg.Service).NotTo(BeNil()) + Expect(dynCfg.Service.Pipelines).To(HaveKey("traces")) + Expect(dynCfg.Service.Pipelines).NotTo(HaveKey("metrics")) + }) + // Regression guards — see comments in config.go for the why behind each. It("uses 'insert' (not 'upsert') on the resource processor so per-datapoint k8s attrs survive", func() { spec := &dbpreview.MonitoringSpec{