Scaling background services causes redis memory explode

[omerblum-cyera]

Bug summary

We're using prefect server on our k8s cluster.

We need to scale our prefect background service, so we split the loop component and the background services.
But, ever since we split them and scaled the background service - the redis memory increasing constantly without anyone evicting it.
Should we set an eviction policy (triming) on the "events" or redis stream?

(Not sure if it's relevant, but "XINFO GROUPS events" returns empty array, so maybe there's an issue with the consuming group?)

Is it a known bug?

It's a matter of time until redis will reach it's limit (which already happened before), which makes the whole separation and scaling the background services impossible.

Version info

Version:              3.6.7
API version:          0.8.4
Python version:       3.12.12
Git commit:           ebfef643
Built:                Thu, Dec 18, 2025 07:57 PM
OS/Arch:              linux/x86_64
Profile:              ephemeral
Server type:          server
Pydantic version:     2.12.5
Server:
  Database:           postgresql
  PostgreSQL version: 16.8
Integrations:
  prefect-redis:      0.2.7

Additional context

No response

[desertaxle]

Thanks for opening an issue @omerblum-cyera! We don't see this behavior in our OSS test bed which runs seperate web and background service deployments, so there might an issue with your set up.

Can you share the settings you configured for your web and background service deployemnts?

[omerblum-cyera]

Thanks for replying so fast!
Here's the background service helm:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: prefect-server-background
  namespace: {{ .Values.prefect.namespace}}
  annotations:
    reloader.stakater.com/auto: "true"
  labels:
    app: prefect-server-background
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prefect-server-background
  template:
    metadata:
      labels:
        app: prefect-server-background
        appClass: prefect-server-background
    spec:
      serviceAccountName: prefect-server
      containers:
      - name: prefect-server-background
        image: prefecthq/prefect:{{ .Values.prefect.imageTag }}
        command:
          - /usr/bin/tini
          - -g
          - --
          - /opt/prefect/entrypoint.sh
        args:
          - prefect
          - server
          - services
          - start
        workingDir: /home/prefect
        envFrom:
        - configMapRef:
            name: prefect-server-config
        volumeMounts:
          - mountPath: /home/prefect
            name: scratch
            subPathExpr: home
          - mountPath: /tmp
            name: scratch
            subPathExpr: tmp
          - mountPath: "/ui_build"
            name: scratch
            subPathExpr: ui-build
          - mountPath: "/health_scripts"
            name: health-scripts
        resources:
          requests:
            memory: {{ .Values.prefect.backgroundService.resources.requests.memory }}
            cpu: {{ .Values.prefect.backgroundService.resources.requests.cpu }}
          limits:
            memory: {{ .Values.prefect.backgroundService.resources.limits.memory }}
      volumes:
        - name: scratch
          emptyDir: {}
        - name: health-scripts
          configMap:
            name: prefect-health-scripts

Here's the configmap:

{{ if .Values.prefect.enabled }}
apiVersion: v1
kind: ConfigMap
metadata:
  name: prefect-server-config
  namespace: {{ .Values.prefect.namespace }}
data:
  HOME: /home/prefect
  PREFECT_API_ENABLE_HTTP2: "true"
  PREFECT_API_MAX_FLOW_RUN_GRAPH_NODES: "500"
  PREFECT_API_URL: /api
  PREFECT_API_ENABLE_METRICS: "true"
  PREFECT_DEBUG_MODE: "false"
  PREFECT_LOGGING_HANDLERS_CONSOLE_FORMATTER: json
  PREFECT_LOGGING_SERVER_LEVEL: WARNING
  PREFECT_LOGGING_TO_API_WHEN_MISSING_FLOW: ignore
  PREFECT_SERVER_API_HOST: 0.0.0.0
  PREFECT_SERVER_API_PORT: "4200"
  PREFECT_UI_ENABLED: "true"
  PREFECT_UI_STATIC_DIRECTORY: /ui_build
  PREFECT_SQLALCHEMY_POOL_SIZE: "10"
  PREFECT_SQLALCHEMY_MAX_OVERFLOW: "20"

  {{- if .Values.prefect.backgroundService.enabled }}
  PREFECT_API_DATABASE_MIGRATE_ON_START: "false"

  PREFECT_MESSAGING_BROKER: "prefect_redis.messaging"
  PREFECT_MESSAGING_CACHE: "prefect_redis.messaging"
  PREFECT_SERVER_EVENTS_CAUSAL_ORDERING: "prefect_redis.ordering"
  PREFECT_SERVER_CONCURRENCY_LEASE_STORAGE: "prefect_redis.lease_storage"
  PREFECT_REDIS_MESSAGING_HOST: {{ .Values.prefect.redis.host | quote }}
  PREFECT_REDIS_MESSAGING_PORT: {{ .Values.prefect.redis.port | quote }}
  PREFECT_REDIS_MESSAGING_DB: "0"
  {{- end }}
{{ end }}

Let me know if you need the api (loop) as well

[desertaxle]

Can you share the web service config too? My hunch is that the web service is writing to a different place than the background services are reading from. You can also take a look at our server Helm chart as a reference.

[omerblum-cyera]

it uses the same config map.
Anyways here's the api yaml:

{{ if .Values.prefect.enabled }}
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prefect-server
  namespace: {{ .Values.prefect.namespace}}
  annotations:
    reloader.stakater.com/auto: "true"
  labels:
    app: prefect-server
spec:
  {{- if .Values.prefect.backgroundService.enabled }}
  replicas: {{ .Values.prefect.replicaCount }}
  {{- else }}
  replicas: 1
  {{- end }}
  template:
    metadata:
      annotations:
        prometheus.io/path: /api/metrics
        prometheus.io/port: {{ .Values.prefect.port | quote }}
        prometheus.io/scrape: "true"
      labels:
        app: prefect-server
        appClass: prefect-server
    spec:
      topologySpreadConstraints:
          - maxSkew: 1
            whenUnsatisfiable: ScheduleAnyway
            topologyKey: topology.kubernetes.io/zone
      serviceAccountName: prefect-server
      containers:
      - name: prefect-server
        image: prefecthq/prefect:{{ .Values.prefect.imageTag }}
        command:
          - /usr/bin/tini
          - -g
          - --
          - /opt/prefect/entrypoint.sh
        args:
          - prefect
          - server
          - start
          {{- if .Values.prefect.backgroundService.enabled }}
          - --no-services
          {{- end }}
        workingDir: /home/prefect
        ports:
        - name: http
          containerPort: {{ .Values.prefect.port }}
        envFrom:
        - configMapRef:
            name: prefect-server-config
        volumeMounts:
          - mountPath: /home/prefect
            name: scratch
            subPathExpr: home
          - mountPath: /tmp
            name: scratch
            subPathExpr: tmp
          - mountPath: "/ui_build"
            name: scratch
            subPathExpr: ui-build
          - mountPath: "/health_scripts"
            name: health-scripts
        readinessProbe:
          httpGet:
            path: /api/ready
            port: {{ .Values.prefect.port }}
          initialDelaySeconds: 10
          periodSeconds: 10
          failureThreshold: 3
          timeoutSeconds: 5
          successThreshold: 1
          initialDelaySeconds: 5
          periodSeconds: 10
          failureThreshold: 10
          timeoutSeconds: 30
          successThreshold: 1
        resources:
          requests:
            memory: {{ .Values.prefect.resources.requests.memory }}
            cpu: {{ .Values.prefect.resources.requests.cpu }}
          limits:
            memory: {{ .Values.prefect.resources.limits.memory }}
      volumes:
        - name: scratch
          emptyDir: {}
        - name: health-scripts
          configMap:
            name: prefect-health-scripts
{{ end }}
{{ if and .Values.prefect.enabled (gt (.Values.prefect.replicaCount | int) 1) }}
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: prefect-server-pdb
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: prefect-server
{{ end }}

[zzstoatzz]

hi @omerblum-cyera - personally i don't see any obvious misconfiguration

could you enable debug logging in the background services?

PREFECT_LOGGING_SERVER_LEVEL: DEBUG

if you don't see the event persister or reactive triggers starting, there might be some silent failure

[omerblum-cyera]

we'll try to use the official chart and I'll update here if it's sorted out or not.
What's the expected size/time for the consumer to consume the events? When should we see the memory used by redis decreasing?

[liran-elhaik-cyera]

We haven’t migrated to the official Helm chart yet, but after further troubleshooting we confirmed that both the server API and background-service pods are configured to use the same Valkey instance (in the same namespace), and basic network connectivity works.

In the background server logs we see Redis/Valkey connection errors (ConnectionRefusedError, NOGROUP when reading streams).

Our current assumption is a startup race condition where Prefect pods start before Valkey is ready. Once the background service fails to connect, it doesn’t appear to recover even after Valkey becomes available, and memory usage continues to grow in the background server, until the pod is restarted.

prefect-background-server.txt

[bdalpe]

@omerblum-cyera @liran-elhaik-cyera can you try with prefect-redis==0.2.8 if you're not already?

This looks quite similar to another issue: #19080 which we added a fix in PR #19817.

[omerblum-cyera]

we're running with 3.6.7-python3.12 prefect image, and the prefect-redis is 0.2.7
Do we need to update the image to 3.6.10? or can we bump only the prefect-redis?

[bdalpe]

@omerblum-cyera I think the easiest way to do this would be to bump the container image. You only need to go up one patch level version to 3.6.8-python3.12 to get prefect-redis==0.2.8

docker run --rm -it prefecthq/prefect:3.6.8-python3.12 -- prefect version
Version:              3.6.8
API version:          0.8.4
Python version:       3.12.12
Git commit:           8ae19ff7
Built:                Wed, Dec 24, 2025 07:28 PM
OS/Arch:              linux/aarch64
Profile:              ephemeral
Server type:          ephemeral
Pydantic version:     2.12.5
Server:
  Database:           sqlite
  SQLite version:     3.50.4
Integrations:
  prefect-redis:      0.2.8

[omerblum-cyera]

we'll try that and update, thanks

[omerblum-cyera]

upgarding to latest image worked, thank you!!