Skip to content

Behavior of sum with .* label_replace breaks when range is larger than 60m #20314

@timonegk

Description

@timonegk

Describe the bug
I am experiencing the following strange bug:

I use a query similar to the following instant query to aggregate data using sum and label_replace:

sum by (component) (
label_replace(
label_replace(
count_over_time({app="alloy"} | logfmt component_id="component_id" [$__range]),
"component", "Other", "component_id", ".*"),
"component", "Prometheus", "component_id", "prometheus.*"))

In reality, the query looks a bit different and has a more label_replace calls, but this is sufficient as an example.
The issue I'm experiencing is that if $__range is at most 60m, I am correctly getting an output of two values ("Other" and "Prometheus"). If I change the range to 61m or anything larger, I am only getting the value for "Other". If I use sum by (component, component_id), the output is the complete in both cases.

To Reproduce
Steps to reproduce the behavior:

  1. Started Loki (3.6.3)
  2. Run the query above on logs that you have (I used alloy above for easier reproducibility, but it can be anything)

Expected behavior
Larger ranges should also contain all target values.

Environment:

  • Infrastructure: Kubernetes
  • Deployment tool: Helm

Screenshots, Promtail config, or terminal output

Image Image

Metadata

Metadata

Assignees

No one assigned

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions