feat(core): add controller log→OTLP bridge (gated OTEL_LOGGING_ENABLED) by braghettos · Pull Request #2151 · kagent-dev/kagent

braghettos · 2026-07-04T16:07:50Z

What

Adds an optional controller log → OTLP bridge: the controller's own zap logs can be exported over OTLP alongside the traces already emitted by InitTracerProvider, enabling log/trace correlation in the controller. This is the third and final observability gap proposed in #2148.

Today go/core/internal/telemetry/ only sets up traces; the controller's zap logs are stdout-only, so they can't be shipped to an OTLP backend or correlated with spans.

How

New internal/telemetry/logging.go:
- InitLoggerProvider builds an OTLP LoggerProvider via autoexport.NewLogExporter (same env-driven exporter/endpoint resolution as InitTracerProvider) and registers it as the global OTel logger provider.
- ControllerZapOpts returns a RawZapOpts(zap.WrapCore(zapcore.NewTee(core, bridgeCore))) option that additively tees the stdout zap core with an otelzap bridge core bound to that provider — stdout logging is preserved, OTLP is added.
tracing.go: extracted the shared newTelemetryResource helper so traces and logs carry identical resource attributes (no behaviour change to InitTracerProvider).
pkg/app/app.go: initialize the logger provider before building the controller logger (so the bridge binds to the configured global provider), then add the tee via ControllerZapOpts.

Gating (default-OFF, additive)

Enabled only when OTEL_LOGGING_ENABLED=true (the flag already registered in pkg/env). When unset, both InitLoggerProvider and ControllerZapOpts are no-ops and the controller logger is byte-identical to today.

Testing

New logging_test.go — asserts ControllerZapOpts() returns no options when disabled, returns a working teed logger when enabled (no panic with the default no-op global provider), and that InitLoggerProvider's shutdown is a safe no-op when disabled.
go build ./core/..., go test -race ./core/internal/telemetry/..., and golangci-lint run all pass.

Notes

Adds go.opentelemetry.io/contrib/bridges/otelzap v0.19.0 (matches the existing otel/log v0.20.0) and promotes otel/log from indirect to direct; no other dependency or version changes.
Opening as draft / work-in-progress per CONTRIBUTING. Companion PRs from [FEATURE] Close three observability gaps: GenAI token metrics, controller reconcile Events, and a controller log→OTLP bridge (all gated default-OFF) #2148: token metrics feat(adk): expose gen_ai.client.token.usage Prometheus metric #2149, reconcile Events feat(core): emit Kubernetes Events on Agent/ModelConfig/MCPServerTool/RemoteMCPServer reconciles #2150.

Refs: #2148

…BLED) Additively tee the controller's zap logger with an otelzap bridge core so the controller's own logs can be exported over OTLP alongside the traces already emitted via InitTracerProvider, without changing stdout logging. - New internal/telemetry/logging.go: - InitLoggerProvider builds an OTLP LoggerProvider via autoexport (same env-driven exporter resolution as InitTracerProvider) and sets it as the global OTel logger provider. - ControllerZapOpts returns a RawZapOpts(zap.WrapCore(NewTee(...))) option that tees the stdout core with an otelzap bridge core bound to that provider. - tracing.go: extract the shared newTelemetryResource helper so traces and logs carry identical resource attributes. - app.go: initialize the logger provider before building the controller logger (so the bridge binds to it) and add the tee via ControllerZapOpts. Gated behind OTEL_LOGGING_ENABLED; when unset both functions are no-ops and the logger is byte-identical to today. Adds otelzap v0.19.0 (matching otel/log v0.20.0); no other dependency changes. Refs: kagent-dev#2148 Signed-off-by: Diego Braga <diego.braga86@gmail.com>

Copilot

Pull request overview

Adds an optional, default-off OpenTelemetry logging pipeline for the controller, enabling controller-runtime zap logs to be exported via OTLP (in addition to existing trace export) for log/trace correlation.

Changes:

Introduces InitLoggerProvider to configure and register a global OTel LoggerProvider via env-driven OTLP autoexport (gated by OTEL_LOGGING_ENABLED).
Adds ControllerZapOpts to tee the controller’s zap core to an otelzap bridge core while preserving stdout logging.
Refactors trace setup to share a common OTel Resource builder so traces and logs carry identical resource attributes.

Reviewed changes

Copilot reviewed 5 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
go/go.mod	Adds `otelzap` bridge dependency and promotes `otel/log` to a direct dependency.
go/go.sum	Updates module checksums for the new/now-direct dependencies.
go/core/pkg/app/app.go	Initializes OTel log provider before constructing the controller logger and applies the zap tee option.
go/core/internal/telemetry/tracing.go	Extracts shared `newTelemetryResource` helper for consistent resource attributes.
go/core/internal/telemetry/logging.go	New OTLP logger provider init + controller zap tee option (gated).
go/core/internal/telemetry/logging_test.go	Adds unit tests for gating behavior and no-op shutdown when disabled.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+	defer func() {
+		shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+		defer cancel()
+		if err := shutdownLogging(shutdownCtx); err != nil {
+			setupLog.Error(err, "failed to shutdown logging")


krisztianfekete

Thanks for the PR!
Added some comments, and just like in #2149, we'd like to see docs/examples to set it up, and some screenshots showing that the changes have been tested in a real setup. An OTel collector + debugexporter is perfect for this purpose if you don't want to ingest the logs into a proper logging backend.

krisztianfekete · 2026-07-07T13:29:43Z

+	if !env.OtelLoggingEnabled.Get() {
+		return nil
+	}
+	bridgeCore := otelzap.NewCore(loggerBridgeName,


I don't think this actually gets us log-trace correlation as-is, which is what you're trying to deliver?

otelzap.Core only adds trace_id/span_id to a record when a context.Context is passed as a zap field. controller-runtime logs through logr -> zapr, and log.FromContext(ctx) only threads values in via WithValues, so I am not sure these records contain spans linked to them.

krisztianfekete · 2026-07-07T13:33:29Z

+	)
+	return []crzap.Opts{
+		crzap.RawZapOpts(zap.WrapCore(func(core zapcore.Core) zapcore.Core {
+			return zapcore.NewTee(core, bridgeCore)


The OTel log SDK puts severity filtering in a processor, not in the exporter or the bridge, so someone running at error level only expects error+ logs, but when they enable this flag it will ship all their info/debug logs to the backend.

We can fix it via https://pkg.go.dev/go.opentelemetry.io/contrib/processors/minsev

krisztianfekete · 2026-07-07T13:36:06Z

+// signal pipelines (traces and logs), keeping their resource attributes
+// identical.
+func newTelemetryResource(ctx context.Context, serviceVersion string) (*resource.Resource, error) {
 	instanceID, err := os.Hostname()


Since newTelemetryResource is now called once by InitTracerProvider and once by InitLoggerProvider, and os.Hostname() falls back to a fresh uuid.New() on failure, it's possible that processes can end up with two different service.instance.ids across the two ttelemetry signals.

Let's build the resource once in app.go and pass the same *resource.Resource into both initializers instead.

krisztianfekete · 2026-07-07T13:40:53Z

+// TestControllerZapOpts_EnabledTeesLogger verifies that when OTEL_LOGGING_ENABLED
+// is set, an otelzap bridge option is returned and the resulting logger is usable
+// (the bridge tees onto the stdout core without panicking).
+func TestControllerZapOpts_EnabledTeesLogger(t *testing.T) {


Let's also test

a record actually having trace_id when there's an active span, and

severity gating vs the stdout level

github-actions Bot added the enhancement New feature or request label Jul 4, 2026

braghettos marked this pull request as ready for review July 4, 2026 16:11

braghettos requested a review from supreme-gg-gg as a code owner July 4, 2026 16:11

Copilot AI review requested due to automatic review settings July 4, 2026 16:11

braghettos requested review from EItanya, ilackarms, iplay88keys, jmhbh and yuval-k as code owners July 4, 2026 16:11

Copilot started reviewing on behalf of braghettos July 4, 2026 16:12 View session

Copilot AI reviewed Jul 4, 2026

View reviewed changes

Comment thread go/core/pkg/app/app.go

Comment on lines +340 to +344

defer func() {

shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second)

defer cancel()

if err := shutdownLogging(shutdownCtx); err != nil {

setupLog.Error(err, "failed to shutdown logging")

braghettos mentioned this pull request Jul 6, 2026

feat(adk): expose gen_ai.client.token.usage Prometheus metric #2149

Open

krisztianfekete suggested changes Jul 7, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(core): add controller log→OTLP bridge (gated OTEL_LOGGING_ENABLED)#2151

feat(core): add controller log→OTLP bridge (gated OTEL_LOGGING_ENABLED)#2151
braghettos wants to merge 1 commit into
kagent-dev:mainfrom
braghettos:observability-log-bridge

braghettos commented Jul 4, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

krisztianfekete left a comment

Uh oh!

krisztianfekete Jul 7, 2026

Uh oh!

krisztianfekete Jul 7, 2026

Uh oh!

krisztianfekete Jul 7, 2026

Uh oh!

krisztianfekete Jul 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

braghettos commented Jul 4, 2026

What

How

Gating (default-OFF, additive)

Testing

Notes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

krisztianfekete left a comment

Choose a reason for hiding this comment

Uh oh!

krisztianfekete Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

krisztianfekete Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

krisztianfekete Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

krisztianfekete Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants