VPA not receiving memory statistics but CPU is updating

[2fst4u]

Which component are you using?: /area vertical-pod-autoscaler

What version of the component are you using?:

Component version: 1.5

What k8s version are you using (kubectl version)?: v1.33.6+k3s1

kubectl version Output

$ kubectl version

What environment is this in?: on prem

What did you expect to happen?: VPAcheckpoint updates the memory bucket weights and subsequently updates the VPA recommendations.

What happened instead?: memory is not updating in the VPAcheckpoints at all, and VPAs are therefore not changing their recommendations.

How to reproduce it (as minimally and precisely as possible): following the readme, clone the repo and run the hack command.

Anything else we need to know?:

This is weird one and I'm not sure how to nail down exactly what's gone wrong. I've actually been running VPAs for years now without much issue until I saw recently that the new inplaceorrecreate option is available, so I decided to give it a try.

I cloned the repo again, ran the hack command, updated all my VPAs to updateMode: "InPlaceOrRecreate" and thought it was working but the memory buckets were frozen in place. After several full deletions of all VPAs and VPAcheckpoints, and after realising I needed the feature flag enabled and trying it with the flag, then removing the CRDs again completely before re-adding them without it, I'm back at square one but my memory statistics don't update at all in my VPAcheckpoints. The sampling is going up and the CPU values appear to be changing but memory does nothing, it stays blank. Here is an example

Name:         open-webui-vpa-open-webui
Namespace:    open-webui
Labels:       <none>
Annotations:  <none>
API Version:  autoscaling.k8s.io/v1
Kind:         VerticalPodAutoscalerCheckpoint
Metadata:
  Creation Timestamp:  2025-12-30T07:45:36Z
  Generation:          73
  Resource Version:    820924750
  UID:                 0b2f3caf-6656-4972-aea3-a1b11af0909e
Spec:
  Container Name:   open-webui
  Vpa Object Name:  open-webui-vpa
Status:
  Cpu Histogram:
    Bucket Weights:
      0:                  10000
      1:                  734
      3:                  145
      4:                  294
      5:                  148
    Reference Timestamp:  2025-12-30T00:00:00Z
    Total Weight:         9.79105681805703
  First Sample Start:     2025-12-30T07:40:20Z
  Last Sample Start:      2025-12-30T08:57:17Z
  Last Update Time:       2025-12-30T08:57:36Z
  Memory Histogram:
    Reference Timestamp:  2025-12-31T00:00:00Z
  Total Samples Count:    77
  Version:                v3
Events:                   <none>

As you can see there are 77 samples and no memory buckets showing. This is the case for all my VPAs now.

I've fully removed the CRDs and re-added them again but I have no idea what's wrong other than maybe a version disparity.

Let me know any commands to show outputs that would help.

[adrianmoisey]

/area vertical-pod-autoscaler

[adrianmoisey]

I can't reproduce this with the instructions you provided. Can you add more details?
What does the VPA resource look like?
Are you using prometheus or metric-server?
Has this worked in the past? Which version of the VPA did it work in?

/area vertical-pod-autoscaler

[2fst4u]

Sorry I fat fingered the close button, please reopen.

I do realise it'll be hard to recreate because I have no idea at what point it stopped working properly.

For that VPA it is:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: open-webui-vpa
  namespace: open-webui
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: StatefulSet
    name: open-webui
  resourcePolicy:
    containerPolicies:
      - containerName: '*'
        minAllowed:
          cpu: 200m
          memory: 0Mi
        maxAllowed:
          cpu: 1
          memory: 4000Mi
        controlledResources:
          - cpu
          - memory
        controlledValues: RequestsAndLimits
  updatePolicy:
    updateMode: "Recreate" 

Built in k3s metrics server which seems to be working fine because pod top shows details.

I can't say which version it was working in sorry because it would have been about 2 years untouched. I've been deploying via the readme "clone" instructions rather than a preferred helm chart or something like that so I haven't had versioning.

[adrianmoisey]

Can you provide a sample Statefulset too?
I'd like to recreate this end to end in Kind

/reopen

[k8s-ci-robot]

@adrianmoisey: Reopened this issue.

Details

In response to this:

Can you provide a sample Statefulset too?
I'd like to recreate this end to end in Kind

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[adrianmoisey]

Can you clarify which mode you're using? I seem to be able to recreate this in InPlaceOrRecreate mode, but not in Recreate mode.

[adrianmoisey]

I believe that the VPA has actually always operated this way. There's a great writeup over here about how the memory histogram works.

[2fst4u]

I believe that the VPA has actually always operated this way. There's a great writeup over here about how the memory histogram works.

The reason I was looking into it in the first place was because I was investigating the statistics to see how long it took for the recommendation to change, so before changing any of this the histogram definitely showed buckets for the memory values for workloads.

[2fst4u]

Can you clarify which mode you're using? I seem to be able to recreate this in InPlaceOrRecreate mode, but not in Recreate mode.

I changed all my VPAs to in inplaceorrecreate, and all my checkpoints appeared to be frozen with bucket weights that wouldn't change. After I wiped the vpas a few times since it appeared broken, I tried to go back to "initial" or "recreate" and neither seem to work now.

So I started in "auto", changed to "inplaceorrecreate", went back to "recreate" and "initial" to see if there was a difference and there isn't, and now no checkpoints show bucket weights. What happens if you change them to one of those modes? Does it keep doing it too?

I can provide a statefulset like requested earlier but this is across all my workloads so I'm not sure any of my manifests are specifically relevant. Let me know if you still want that.

[2fst4u]

Further to add, the absence of the memory buckets isn't the main concern, I believe this is in turn causing my VPAs to not actually change their recommendations. For example in the above VPA the current state is

Name:         open-webui-vpa
Namespace:    open-webui
Labels:       kustomize.toolkit.fluxcd.io/name=apps
              kustomize.toolkit.fluxcd.io/namespace=flux-system
Annotations:  <none>
API Version:  autoscaling.k8s.io/v1
Kind:         VerticalPodAutoscaler
Metadata:
  Creation Timestamp:  2025-12-30T07:44:39Z
  Generation:          3
  Resource Version:    821218535
  UID:                 2a13b1c2-9c5e-44d8-a8fe-98acd9d5d9c7
Spec:
  Resource Policy:
    Container Policies:
      Container Name:  *
      Controlled Resources:
        cpu
        memory
      Controlled Values:  RequestsAndLimits
      Max Allowed:
        Cpu:     1
        Memory:  4000Mi
      Min Allowed:
        Cpu:     200m
        Memory:  0Mi
  Target Ref:
    API Version:  apps/v1
    Kind:         StatefulSet
    Name:         open-webui
  Update Policy:
    Update Mode:  Auto
Status:
  Conditions:
    Last Transition Time:  2025-12-30T07:45:36Z
    Status:                True
    Type:                  RecommendationProvided
  Recommendation:
    Container Recommendations:
      Container Name:  open-webui
      Lower Bound:
        Cpu:     200m
        Memory:  862739890
      Target:
        Cpu:     200m
        Memory:  1238659775
      Uncapped Target:
        Cpu:     25m
        Memory:  1238659775
      Upper Bound:
        Cpu:     200m
        Memory:  3531294834
Events:          <none>

Providing a recommendation of approx 1.2Gi, but the current (and steady historically for the past day) usage of the pod is

kubectl top pod -n open-webui
NAME                                CPU(cores)   MEMORY(bytes)
open-webui-0                        2m           687Mi

Remembering I said above that I've deleted the VPAs and their checkpoints several times over in the past day, so the only usage it should hold should be where the pod has settled lately or thereabouts. No matter what I do the recommendations for memory remain the same but the CPU recommendations appear to be changing as intended.