Payload Too Big

[syedg1]

Our metrics pipeline is setup such that the datadog agent forwards metrics to vector, but sends traces and logs directly to datadog (ie. enabled the metrics observability pipeline worker and configured it to vector).

Currently, I am load testing the metrics portion of the pipeline using Lading to emit dogstatsd data to the datadog agent via UDS socket to ensure that no data is being lost when forwarding to vector. However, I noticed that the following logs, which coincide with spikes in the datadog.agent.point.dropped internal metric.

2025-10-30 23:33:19 UTC | CORE | ERROR | (comp/forwarder/defaultforwarder/transaction/transaction.go:431 in internalProcess) | Error code "413 Payload Too Large" received while sending transaction to "https://7-71-2-app.agent.datadoghq.com./intake/": "Payload too big", dropping it

Is there any way to filter by the type of data being dropped (ie. metric, trace, log), so that we can configure our alerts to only monitor for dropped metrics?

[Babbili]

hey @syedg1 is the payload body available ?, if it's available you can create an alert for the payload body pattern that's related to metrics plus the error you're having.

Another way also maybe helpful if you know which metric its data is being dropped is to create a Metric alert for it on certain value, you may identify first through monitoring which value of the metric its data being dropped after and set an alert on that value.

if this is useful to answer your question, please mark it as answered, so we close the issue, thank you for your question

[syedg1]

@Babbili I've enabled payload logging, but it's difficult to decipher which payload is causing the issues.

Are you able to share which metrics can be used to monitor dogstatsd dropped metrics? I've been looking at the datadog.dogstatsd.client.packets_dropped metric which does not show anything being dropped despite us noticing dropped metrics. However, I am not sure if these drops are happening on the Datadog agent or the Vector aggregator as both do not show any drops in the relevant internal metrics.

[itxashancode]

The logs show transaction type likely for metrics? The forwarder logs for metrics, traces, logs? The internal metric datadog.agent.point.dropped counts dropped points (metrics). Need to check if we can tag dropped events by type. Look at agent code: forwarder/transaction.go internalProcess logs error "Payload too big" and drops transaction. The transaction includes payload type? Might have field like "kind" or "payload_type". In agent config, there is logs_enabled, apm_enabled, process_agent_enabled. The forwarder likely has separate forwarders for metrics, traces, logs. The error is from defaultforwarder? The log shows comp/forwarder/defaultforwarder/transaction/... So defaultforwarder handles all? Actually defaultforwarder likely for metrics only? But we can create metric alert on datadog.agent.point.dropped with tag type:metrics? The internal metric may have tags: type:point