feature: Add W3C trace context support via adding the traceparent header from $http_traceparent (for OpenTracing + OpenTelemetry)

[sriemer]

http: Add W3C trace context support to send_request()

NGINX can be used together with an NGINX tracer with W3C trace context
support. That tracer can be based on OpenTracing or OpenTelemetry
(OTel), such as the nginx-otel module.
In the Observability space it is required that all HTTP request
tracers in the line do trace propagation or trace forwarding at least.
This is usually done by sending out an updated traceparent header.
It contains for example a trace ID and a span ID so that the
Observability backend such as Jaeger can correlate all spans to the
same trace.

To be able to trace HTTP requests sent from regular NGINX code and
also the ones sent by LUA HTTP client code with the same NGINX tracer,
the LUA HTTP client code has to be extended to add the traceparent
header from ngx.var.http_traceparent. The value of this
$http_traceparent variable is set by the NGINX tracer. It
updates the span ID inside the traceparent value if the NGINX tracer
is set to propagation mode.
Example with nginx-otel config: otel_trace_context propagate;.

So extend the send_request() function for this, because it already
adds other headers as well and is used when calling request_uri()
as well.

Check if the traceparent header is not set yet but
ngx.var.http_traceparent is set and set it from there.
Also add a code comment describing this as well.

References:

• https://www.w3.org/TR/trace-context/
• https://github.com/nginxinc/nginx-otel
• https://nginx.org/en/docs/ngx_otel_module.html
• https://nginx.org/en/docs/http/ngx_http_core_module.html#var_http_

---

t: Add 3 traceparent header tests

The new traceparent header auto-injection from
ngx.var.http_traceparent has to be tested.

So add a new file t/21-traceparent-header.t containing 3 tests
for that:

• TEST 1: No traceparent header is set
• TEST 2: The traceparent header is correctly added when
  ngx.var.http_traceparent is used
• TEST 3: The traceparent header is not modified from
  ngx.var.http_traceparent if it is already set

Use an NGINX directive set $http_traceparent '00-000...-01'; in
front of the LUA block to emulate nginx-otel behavior.

The case that there is a traceparent header added with
request_uri() and ngx.var.http_traceparent is not set, is already
covered by tests 1 and 3.

[sriemer]

Build fails here due to deprecated Luarocks. See #333.
All code has been tested against OpenResty 1.29.2.2. I've built the test environment from a custom Dockerfile.
See #341.

[sriemer]

The nginx.conf parts I've used to test the real thing together with tcpdumping in a docker-compose based demo:

load_module modules/ngx_otel_module.so;
...
http {
...
    upstream backend {
      server server-app:8080;
    }
    otel_exporter {
        endpoint myobservability-agent:4317;
    }
    otel_trace on;
    otel_service_name myname-openresty-otel;
    server {
        ...
        location /openresty-otel-demo {
            otel_trace_context propagate;
            proxy_pass http://backend;
        }
        location /lua-otel-demo {
            otel_trace_context propagate;
            content_by_lua_block {
                local http = require "resty.http"
                local httpc = http.new()
                local res, err = httpc:request_uri("http://server-app:8080", {
                    method = "GET",
                })
                if not res then
                    ngx.status = ngx.HTTP_BAD_GATEWAY
                    ngx.header["Content-Type"] = "text/plain; charset=utf-8"
                    ngx.say("failed to call upstream: ", err)
                    return
                end
                ngx.status = res.status
                if res.headers["Content-Type"] then
                    ngx.header["Content-Type"] = res.headers["Content-Type"]
                end
                ngx.say(res.body)
            }
        }
    }
}

[sriemer]

@pintsized Please review. TIA

Also let me know if you need help from an experienced FOSS maintainer like me in reviewing open PRs.

[sriemer]

@pintsized Any update here? TIA

[sriemer]

@pintsized Do you still have any motivation to continue this SW project? Do you need more help? TIA

[sriemer]

This even works with OpenTracing based tracers as well which set a traceparent header.
So works with all NGINX tracers setting a traceparent header.

[pintsized]

@pintsized Do you still have any motivation to continue this SW project? Do you need more help? TIA

@sriemer sorry for the painfully slow response. Yes, the problem is that I really do not have time to maintain this project, or the other related projects in this org. The org itself (ledgetech) was created with the aspiration that perhaps one day other maintainers could look after these modules - would you be interested if I added you to the org?

[sriemer]

Would you be interested if I added you to the org?

@pintsized Sure, would definitely avoid that we have to go with a fork that we would have to maintain anyway and have to convince others to move over. Keeping the known upstream definitely helps a lot. I can definitely help the other contributors to get their parts in as well if quality is right. Thanks in advance for adding me.

[sriemer]

Rebased for latest CI checks.

[sriemer]

Force-pushed to change commit description of commit 1 and the code comment to be working with any NGINX tracer supporting the traceparent header. So this also covers the ones based on OpenTracing. Only using nginx-otel as an example NGINX tracer now.
New code comment: -- OpenTelemetry support with NGINX tracer

[sriemer]

Rebased for PR #343 changes.

[sriemer]

@aravindjayanthi98 Amended that comment, the commit description of the first commit and long and short PR descriptions. Added another reference to https://www.w3.org/TR/trace-context/.
Please rereview. TIA

[aravindjayanthi98]

LGTM!

[sriemer]

Merged with a merge commit to keep a link to the conversation here because it contains important information, such as the NGINX config in #340 (comment).
Thanks. 😃