Kubernetes: run telemetry has empty host — TRIGGER_WORKER_INSTANCE_NAME (spec.nodeName) not forwarded into the run process

[brentshulman-silkline]

Summary

On self-hosted Kubernetes, task-run spans and logs exported over OTLP to an off-node collector (e.g. a centralized OpenTelemetry/Datadog Agent) arrive with no host. Datadog tags every such span issue_type:empty_hostname. There is currently no supported way for a user's trigger.config.ts telemetry resource to recover the node, because the node identity never reaches the run process — even though the supervisor already puts it on the pod.

Root cause

The Kubernetes workload manager already injects the node name into the run-controller container via the downward API — apps/supervisor/src/workloadManager/kubernetes.ts:

{
  name: "TRIGGER_WORKER_INSTANCE_NAME",
  valueFrom: { fieldRef: { fieldPath: "spec.nodeName" } },
},

But the actual task-run worker is fork()ed with an explicit, replaced env (not the container's process.env):

• packages/cli-v3/src/entryPoints/managed/taskRunProcessProvider.ts → buildProcessEnvironment() builds the child env as { ...taskRunEnv, ...this.env.gatherProcessEnv(), TRIGGER_MACHINE_ID, HEARTBEAT_INTERVAL_MS }.
• packages/cli-v3/src/executions/taskRunProcess.ts → fork(..., { env: fullEnv }) where fullEnv = { ...$env, OTEL_IMPORT_HOOK_INCLUDES, NODE_OPTIONS, PATH, TRIGGER_PROCESS_FORK_START_TIME, TRIGGER_WARM_START, TRIGGERDOTDEV }.
• packages/cli-v3/src/entryPoints/managed/env.ts → RunnerEnv.gatherProcessEnv() returns a fixed allowlist: NODE_ENV, NODE_EXTRA_CA_CERTS, OTEL_EXPORTER_OTLP_ENDPOINT, TRIGGER_OTEL_EXPORTER_OTLP_ENDPOINT, UV_USE_IO_URING.

So in the run process, process.env.TRIGGER_WORKER_INSTANCE_NAME (and HOSTNAME) are undefined. The node name exists on the pod but is dropped before it can be attached to the OTel resource built from config.telemetry.resource.

For comparison, the Docker workload manager already forwards it to the runner (apps/supervisor/src/workloadManager/docker.ts): `TRIGGER_WORKER_INSTANCE_NAME=${env.TRIGGER_WORKER_INSTANCE_NAME}` — so k8s is the inconsistent path here.

Requested change (one line)

Forward TRIGGER_WORKER_INSTANCE_NAME from the run process env allowlist. It is already in the Env schema and has a getter (get TRIGGER_WORKER_INSTANCE_NAME()), so this is additive and safe (the existing undefined filter drops it when unset).

File: packages/cli-v3/src/entryPoints/managed/env.ts, method RunnerEnv.gatherProcessEnv()

     const $env = {
       NODE_ENV: this.NODE_ENV,
       NODE_EXTRA_CA_CERTS: this.NODE_EXTRA_CA_CERTS,
       OTEL_EXPORTER_OTLP_ENDPOINT: this.OTEL_EXPORTER_OTLP_ENDPOINT,
       TRIGGER_OTEL_EXPORTER_OTLP_ENDPOINT: this.OTEL_EXPORTER_OTLP_ENDPOINT,
       UV_USE_IO_URING: this.UV_USE_IO_URING,
+      // Node the run is executing on (spec.nodeName on k8s). Forwarding it lets
+      // trigger.config.ts telemetry set an OTel host.name so run traces/logs
+      // exported to an off-node collector are attributed to a host.
+      TRIGGER_WORKER_INSTANCE_NAME: this.TRIGGER_WORKER_INSTANCE_NAME,
     };

With that, a user can map it in trigger.config.ts:

const nodeName = process.env.TRIGGER_WORKER_INSTANCE_NAME; // = spec.nodeName on k8s
export default defineConfig({
  telemetry: {
    resource: resourceFromAttributes(
      nodeName ? { "host.name": nodeName, "k8s.node.name": nodeName } : {}
    ),
    // ...
  },
});

which clears empty_hostname and attributes each run to its actual node.

Scope / risk

• Additive: forwards one already-present, non-secret identity variable.
• No behavior change when unset (existing undefined filter).
• Aligns the k8s workload manager with the Docker one, which already forwards it.

Notes

I have the exact change (plus a changeset) staged on a branch and am happy to open the PR once vouched — filing this as an issue first per the contributor policy. Reference branch/diff: PR #4213 (auto-closed by the unvouched-author check).