[Feature]: Support max_recv_msg_size_mib for gRPC storage client

[nico34638]

Requirement

We have your traces/spans. For internal reasons, we need to be able to display them in Jaeger. We use grpc backend storage.

Problem

Remote gRPC storage fails with large responses:

rpc error: code = ResourceExhausted desc = grpc: received message larger than max (10340064 vs. 4194304)

The gRPC client is limited to 4MB but OpenTelemetry's configgrpc.ClientConfig doesn't expose max_recv_msg_size_mib (only available for servers).

Proposal

Add max_recv_msg_size_mib config option for gRPC storage backends to allow configuring client-side max receive message size, similar to what exists for OTLP receivers.

Open questions

No response

[yurishkuro]

Remote gRPC storage fails with large responses

Why is this a problem with Jaeger if the error is coming from external gRPC storage?

[nico34638]

The problem lies in the Jaeger GRPC client, which contacts the external storage that is limited by default to 4MB, whereas my external storage has a higher maximum capacity.

[nico34638]

I think we could add an option in the configuration and use it when creating the grpc client in the factory internal/storage/v2/grpc/factory.go

[yurishkuro]

We're already using OTEL's configgrpc.ClientConfig for gRPC storage. If max message size needs to be supported it should be added there.

[nico34638]

I agree, I would like to add the following option max_recv_msg_size_mib: 15

Actually config

    backends:
      quickwit_trace_storage:
        grpc:
          endpoint: "my-backend:443"

Config with new option:

    backends:
      quickwit_trace_storage:
        grpc:
          endpoint: "my-backend:443"
          max_recv_msg_size_mib: 150

[yurishkuro]

you will need to open a ticket in OTEL if such option is not supported yet