Thanos-query OOMs

[evilr00t]

After upgrading from v0.39.2 to v0.40.1 I've noticed that thanos-query service is being OOM killed every couple of minutes, I've bumped memory from 8GiB to 12GiB but it didn't help.

Didn't change any options during the upgrade.

Red line represents OOM Kill event.

[GiedriusS]

Checking, noticed it too.

[GiedriusS]

@evilr00t do you have gRPC compression enabled?

[evilr00t]

@GiedriusS No. We use default options for thanos-query.

[evilr00t]

Looks like CPU usage also is pretty high. We've rolled back to v0.39.2 for now.

[GiedriusS]

@evilr00t could you paste your command-line parameters?

[evilr00t]

@GiedriusS
We use Bitnami Helm Chart with official Thanos images - we don't use Bitnami images, only helm chart templates.

This is the additional parameter we use for thanos-query.

                extraFlags = [
                  "--store.limits.request-series=300000"
                ]

And in the pod the arguments looks like:

    Args:
      query
      --log.level=info
      --log.format=logfmt
      --grpc-address=0.0.0.0:10901
      --http-address=0.0.0.0:10902
      --query.replica-label=thanos_replica
      --endpoint-group=thanos-storegateway-headless.default.svc.cluster.local:10901
      --endpoint=dnssrv+_grpc._tcp.thanos-ruler.default.svc.cluster.local
      --endpoint=dnssrv+_grpc._tcp.thanos-receive-headless.default.svc.cluster.local
      --alert.query-url=https://thanos.monitoring.FOOBAR.com/
      --store.limits.request-series=300000

[yeya24]

Would you be able to attach any heap dump before the OOM kill?

[chalut01]

me too.

[GiedriusS]

Could you please us out here by providing pprof outputs? When this happens, could you go to http://querierIP:querierHTTPPort/debug/pprof and attach heap, profile, allocs outputs (the binary files)? So far I've been unable to pinpoint the cause.

[evilr00t]

allocs: https://pprof.me/21e10534836f76789c044f64b4527853/?profileType=profile%3Aalloc_objects%3Acount%3Aspace%3Abytes

heap: https://pprof.me/9d94a181f875d78adf392417f9b50430/?profileType=profile%3Aalloc_objects%3Acount%3Aspace%3Abytes

profile: https://pprof.me/b6eb381a23bec112e9bf3b9d6d52c064/?profileType=profile%3Acpu%3Ananoseconds%3Acpu%3Ananoseconds%3Adelta

@GiedriusS @yeya24

[GiedriusS]

First we need something like this: grpc/grpc-go#8697. But I think I will just make a fork while it is not merged.

[GiedriusS]

I will test that ☝️ a bit more and if everything is ok then I will release a new version with a fix.

[tomfankhaenel]

We also saw the same behavior after the update to v0.40.1

[GiedriusS]

Could someone please try the newest main on one pod and report results?

[Tarot67]

I have same issue. I try to build main from repo i will test soon as possible