haproxy reloading every minute

[alexraitz-ship-it]

k8s: AKS, version 1.33.8
chart: kubernetes-ingress version 1.52.0

I have installed kubernetes-ingress via helm (namespace haproxy). The deployment is successful.

$ helm list -n haproxy
NAME    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                           APP VERSION
haproxy haproxy         1               2026-06-15 08:24:25.8675661 -0400 EDT   deployed        kubernetes-ingress-1.52.0       3.2.9

$ kk get all -n haproxy
NAME                                             READY   STATUS    RESTARTS   AGE
pod/haproxy-kubernetes-ingress-5c89b5899-2dwdn   1/1     Running   0          13m
pod/haproxy-kubernetes-ingress-5c89b5899-zh8d6   1/1     Running   0          13m

NAME                                 TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)                                                                  AGE
service/haproxy-kubernetes-ingress   LoadBalancer   10.0.154.143   10.253.2.10   80:31644/TCP,443:31658/TCP,443:31658/UDP,1024:30204/TCP,6060:31156/TCP   13m

NAME                                         READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/haproxy-kubernetes-ingress   2/2     2            2           13m

NAME                                                   DESIRED   CURRENT   READY   AGE
replicaset.apps/haproxy-kubernetes-ingress-5c89b5899   2         2  

I have an ingress configured for argocd. It uses an external Azure load balancer with a private IP:

$ kk get ingress -n argocd -o yaml
apiVersion: v1
items:
- apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    annotations:
      haproxy.org/ssl-passthrough: "true"
      meta.helm.sh/release-name: argocd
      meta.helm.sh/release-namespace: argocd
      service.beta.kubernetes.io/azure-load-balancer-internal: "true"
      service.beta.kubernetes.io/azure-load-balancer-ipv4: 10.253.2.10
    creationTimestamp: "2026-06-12T18:27:32Z"
    generation: 1
    labels:
      app.kubernetes.io/managed-by: Helm
    name: argocd-server-ingress
    namespace: argocd
    resourceVersion: "39788614"
    uid: b5ae6186-257f-4a6e-98e3-e5997676df79
  spec:
    ingressClassName: haproxy
    rules:
    - host: <redacted>
      http:
        paths:
        - backend:
            service:
              name: argocd-server
              port:
                number: 443
          path: /
          pathType: Prefix
  status:
    loadBalancer:
      ingress:
      - ip: 10.253.2.10
kind: List
metadata:
  resourceVersion: "" 

My values.yaml:

controller:
  service:
    type: LoadBalancer
    loadBalancerIP: "10.253.2.10"
    annotations:
      service.beta.kubernetes.io/azure-load-balancer-internal: "true"
      service.beta.kubernetes.io/azure-load-balancer-ipv4: "10.253.2.10"
      service.beta.kubernetes.io/azure-load-balancer-internal-subnet: "<redacted>"

The external load balancer is deployed and healthy.

There is no umbrella chart, custom configMap, custom CRDs, etc.

Haproxy reloads every minute as follows:

2026/06/15 12:44:07 INFO    controller/global.go:91 [transactionID=00c5f28d-afd8-477a-9699-06c8ca6bf243] reload required : Global config updated: map[LogTargetList:[[] [0x16314de223c0]]]
[NOTICE]   (69) : Reloading HAProxy
[NOTICE]   (69) : Initializing new worker (685)
[NOTICE]   (685) : haproxy version is 3.2.19-06868cd6e
[WARNING]  (685) : config : parsing [/etc/haproxy/haproxy.cfg:77] : defaults section 'haproxytech' (declared at /etc/haproxy/haproxy.cfg:20) is explicitly referenced by another proxy and implicitly used here. To avoid any ambiguity don't mix both usage. Add a last defaults section not explicitly used or always use explicit references.
[WARNING]  (685) : config : parsing [/etc/haproxy/haproxy.cfg:90] : defaults section 'haproxytech' (declared at /etc/haproxy/haproxy.cfg:20) is explicitly referenced by another proxy and implicitly used here. To avoid any ambiguity don't mix both usage. Add a last defaults section not explicitly used or always use explicit references.
[WARNING]  (685) : config : parsing [/etc/haproxy/haproxy.cfg:102] : defaults section 'haproxytech' (declared at /etc/haproxy/haproxy.cfg:20) is explicitly referenced by another proxy and implicitly used here. To avoid any ambiguity don't mix both usage. Add a last defaults section not explicitly used or always use explicit references.
[WARNING]  (685) : config : parsing [/etc/haproxy/haproxy.cfg:150] : defaults section 'haproxytech' (declared at /etc/haproxy/haproxy.cfg:20) is explicitly referenced by another proxy and implicitly used here. To avoid any ambiguity don't mix both usage. Add a last defaults section not explicitly used or always use explicit references.
[WARNING]  (685) : config : parsing [/etc/haproxy/haproxy.cfg:199] : defaults section 'haproxytech' (declared at /etc/haproxy/haproxy.cfg:20) is explicitly referenced by another proxy and implicitly used here. To avoid any ambiguity don't mix both usage. Add a last defaults section not explicitly used or always use explicit references.
[WARNING]  (685) : log format ignored for frontend 'healthz' since it has no log address.
[WARNING]  (685) : log format ignored for frontend 'stats' since it has no log address.
[NOTICE]   (69) : Loading success.
[WARNING]  (676) : Proxy healthz stopped (cumulated conns: FE: 3, BE: 0).
[WARNING]  (676) : Proxy http stopped (cumulated conns: FE: 6, BE: 0).
[WARNING]  (676) : Proxy ssl stopped (cumulated conns: FE: 6, BE: 0).
[WARNING]  (676) : Proxy stats stopped (cumulated conns: FE: 8, BE: 0).
[WARNING]  (676) : Proxy https stopped (cumulated conns: FE: 6, BE: 0).
[WARNING]  (676) : Proxy argocd_svc_argocd-server_https stopped (cumulated conns: FE: 0, BE: 0).
[WARNING]  (676) : Proxy haproxy_svc_default-local-service_http stopped (cumulated conns: FE: 0, BE: 12).
[WARNING]  (676) : Proxy ssl-backend stopped (cumulated conns: FE: 0, BE: 6).
2026/06/15 12:44:07 INFO    controller/controller.go:229 [transactionID=00c5f28d-afd8-477a-9699-06c8ca6bf243] HAProxy reloaded
[NOTICE]   (69) : haproxy version is 3.2.19-06868cd6e
[WARNING]  (69) : Former worker (676) exited with code 0 (Exit)

Expected behavior: A clean deployment should not reload every minute.

[dkorunic]

@alexraitz-ship-it Can you show complete logs for at least 15-20 minutes with several reloads? Log above shows a single transaction and a single reload.

[alexraitz-ship-it]

Sure thing! It actually looks like every 5s.

haproxy-logs.txt

[alexraitz-ship-it]

I noticed that the argocd helm chart uses haproxy with redis, though not via kubnernetes-ingress. Could this possibly be related? Here is the configmap snippet:

haproxy.cfg:
----
defaults REDIS
  mode tcp
  timeout connect 4s
  timeout server 330s
  timeout client 330s
  timeout check 2s

listen health_check_http_url
  bind [::]:8888  v4v6
  mode http
  monitor-uri /healthz
  option      dontlognull
# Check Sentinel and whether they are nominated master
backend check_if_redis_is_master_0
  mode tcp
  option tcp-check
  tcp-check connect
  tcp-check send PING\r\n
  tcp-check expect string +PONG
  tcp-check send SENTINEL\ get-master-addr-by-name\ argocd\r\n
  tcp-check expect string REPLACE_ANNOUNCE0
  tcp-check send QUIT\r\n
  server R0 argocd-redis-ha-announce-0:26379 check inter 1s
  server R1 argocd-redis-ha-announce-1:26379 check inter 1s
  server R2 argocd-redis-ha-announce-2:26379 check inter 1s
# Check Sentinel and whether they are nominated master
backend check_if_redis_is_master_1
  mode tcp
  option tcp-check
  tcp-check connect
  tcp-check send PING\r\n
  tcp-check expect string +PONG
  tcp-check send SENTINEL\ get-master-addr-by-name\ argocd\r\n
  tcp-check expect string REPLACE_ANNOUNCE1
  tcp-check send QUIT\r\n
  server R0 argocd-redis-ha-announce-0:26379 check inter 1s
  server R1 argocd-redis-ha-announce-1:26379 check inter 1s
  server R2 argocd-redis-ha-announce-2:26379 check inter 1s
# Check Sentinel and whether they are nominated master
backend check_if_redis_is_master_2
  mode tcp
  option tcp-check
  tcp-check connect
  tcp-check send PING\r\n
  tcp-check expect string +PONG
  tcp-check send SENTINEL\ get-master-addr-by-name\ argocd\r\n
  tcp-check expect string REPLACE_ANNOUNCE2
  tcp-check send QUIT\r\n
  server R0 argocd-redis-ha-announce-0:26379 check inter 1s
  server R1 argocd-redis-ha-announce-1:26379 check inter 1s
  server R2 argocd-redis-ha-announce-2:26379 check inter 1s

# decide redis backend to use
#master
frontend ft_redis_master
  bind [::]:6379 v4v6
  use_backend bk_redis_master
# Check all redis servers to see if they think they are master
backend bk_redis_master
  mode tcp
  option tcp-check
  tcp-check connect
  tcp-check send "AUTH ${AUTH}"\r\n
  tcp-check expect string +OK
  tcp-check send PING\r\n
  tcp-check expect string +PONG
  tcp-check send info\ replication\r\n
  tcp-check expect string role:master
  tcp-check send QUIT\r\n
  tcp-check expect string +OK
  use-server R0 if { srv_is_up(R0) } { nbsrv(check_if_redis_is_master_0) ge 2 }
  server R0 argocd-redis-ha-announce-0:6379 check inter 1s fall 1 rise 1
  use-server R1 if { srv_is_up(R1) } { nbsrv(check_if_redis_is_master_1) ge 2 }
  server R1 argocd-redis-ha-announce-1:6379 check inter 1s fall 1 rise 1
  use-server R2 if { srv_is_up(R2) } { nbsrv(check_if_redis_is_master_2) ge 2 }
  server R2 argocd-redis-ha-announce-2:6379 check inter 1s fall 1 rise 1
frontend stats
  mode http
  bind [::]:9101 v4v6
  http-request use-service prometheus-exporter if { path /metrics }
  stats enable
  stats uri /stats
  stats refresh 10s

[alexraitz-ship-it]

I did update the argocd deployment to add a default and global config. Unfortunately the issue persists.

[alexraitz-ship-it]

Here are the contents of /etc/haproxy/haproxy.cfg on the leader pod:

global
  default-path config
  master-worker
  pidfile /var/run/haproxy.pid
  stats socket /var/run/haproxy-runtime-api.sock expose-fd listeners level admin
  stats timeout 36000
  limited-quic
  ssl-default-bind-options no-sslv3 no-tls-tickets no-tlsv10
  ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA:!3DES
  hard-stop-after 1800000
  log 127.0.0.1 local0 notice

defaults haproxytech
  log global
  log-format '%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs "%HM %[var(txn.base)] %HV"'
  option dontlognull
  option http-keep-alive
  timeout http-request 5000
  timeout connect 5000
  timeout client 50000
  timeout queue 5000
  timeout server 50000
  timeout tunnel 3600000
  timeout http-keep-alive 60000

peers localinstance
  peer haproxy-kubernetes-ingress-5f574b575f-xzbf6 10.253.5.86:10000

frontend healthz from haproxytech
  mode http
  bind 0.0.0.0:1042 name v4
  bind :::1042 name v6 v4v6
  monitor-uri /healthz
  no log

frontend http from haproxytech
  mode http
  bind 0.0.0.0:8080 name v4
  bind [::]:8080 name v6
  http-request set-var(txn.base) base
  http-request set-var(txn.path) path
  http-request set-var(txn.host) req.hdr(Host),field(1,:),lower
  http-request set-var(txn.host_match) var(txn.host),map(/etc/haproxy/maps/host.map)
  http-request set-var(txn.host_match) var(txn.host),regsub(^[^.]*,,),map(/etc/haproxy/maps/host.map,'') if !{ var(txn.host_match) -m found }
  http-request set-var(txn.path_match) var(txn.host_match),concat(,txn.path,),map(/etc/haproxy/maps/path-exact.map)
  http-request set-var(txn.path_match) var(txn.host_match),concat(,txn.path,),map(/etc/haproxy/maps/path-prefix-exact.map) if !{ var(txn.path_match) -m found }
  http-request set-var(txn.path_match) var(txn.host_match),concat(,txn.path,),map_beg(/etc/haproxy/maps/path-prefix.map) if !{ var(txn.path_match) -m found }
  use_backend %[var(txn.path_match),field(1,.)]
  default_backend haproxy_svc_default-local-service_http

frontend https from haproxytech
  mode http
  bind quic6@[::]:8443 name quicv6 crt /etc/haproxy/certs/frontend ssl alpn h3
  bind unix@/var/run/ssl-frontend.sock name unixsock crt /etc/haproxy/certs/frontend ssl alpn h2,http/1.1 accept-proxy
  bind quic4@0.0.0.0:8443 name quicv4 crt /etc/haproxy/certs/frontend ssl alpn h3
  http-request set-var(txn.base) base
  http-request set-var(txn.path) path
  http-request set-var(txn.host) req.hdr(Host),field(1,:),lower
  http-request set-var(txn.host_match) var(txn.host),map(/etc/haproxy/maps/host.map)
  http-request set-var(txn.host_match) var(txn.host),regsub(^[^.]*,,),map(/etc/haproxy/maps/host.map,'') if !{ var(txn.host_match) -m found }
  http-request set-var(txn.path_match) var(txn.host_match),concat(,txn.path,),map(/etc/haproxy/maps/path-exact.map)
  http-request set-var(txn.path_match) var(txn.host_match),concat(,txn.path,),map(/etc/haproxy/maps/path-prefix-exact.map) if !{ var(txn.path_match) -m found }
  http-request set-var(txn.path_match) var(txn.host_match),concat(,txn.path,),map_beg(/etc/haproxy/maps/path-prefix.map) if !{ var(txn.path_match) -m found }
  http-request redirect scheme https unless { ssl_fc }
  http-request set-header X-Forwarded-Proto https
  use_backend %[var(txn.path_match),field(1,.)]
  default_backend haproxy_svc_default-local-service_http
  http-response set-header alt-svc "h3=\":443\";ma=60;"

frontend ssl
  mode tcp
  bind 0.0.0.0:8443 name v4
  bind [::]:8443 name v6 v4v6
  log-format '%ci:%cp [%t] %ft %b/%s %Tw/%Tc/%Tt %B %ts %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs SNI: %[var(sess.sni)]'
  tcp-request content reject if !{ req_ssl_hello_type 1 }
  tcp-request inspect-delay 5000
  tcp-request content set-var(sess.sni) req_ssl_sni
  tcp-request content set-var(txn.sni_match) req_ssl_sni,map(/etc/haproxy/maps/sni.map)
  tcp-request content set-var(txn.sni_match) req_ssl_sni,regsub(^[^.]*,,),map(/etc/haproxy/maps/sni.map)
  use_backend %[var(txn.sni_match),field(1,.)]
  default_backend ssl-backend

frontend stats from haproxytech
  mode http
  bind :::1024 name v6
  bind *:1024 name stats
  no log
  stats enable
  stats show-legends
  stats refresh 10000
  stats uri /
  http-request set-var(txn.base) base
  http-request use-service prometheus-exporter if { path /metrics }

backend argocd_svc_argocd-server_https from haproxytech
  mode tcp
  balance roundrobin
  no option abortonclose
  default-server check
  server SRV_1 10.253.5.66:8080 enabled
  server SRV_2 10.253.5.90:8080 enabled
  server SRV_3 127.0.0.1:1 disabled
  server SRV_4 127.0.0.1:1 disabled
  server SRV_5 127.0.0.1:1 disabled
  server SRV_6 127.0.0.1:1 disabled
  server SRV_7 127.0.0.1:1 disabled
  server SRV_8 127.0.0.1:1 disabled
  server SRV_9 127.0.0.1:1 disabled
  server SRV_10 127.0.0.1:1 disabled
  server SRV_11 127.0.0.1:1 disabled
  server SRV_12 127.0.0.1:1 disabled
  server SRV_13 127.0.0.1:1 disabled
  server SRV_14 127.0.0.1:1 disabled
  server SRV_15 127.0.0.1:1 disabled
  server SRV_16 127.0.0.1:1 disabled
  server SRV_17 127.0.0.1:1 disabled
  server SRV_18 127.0.0.1:1 disabled
  server SRV_19 127.0.0.1:1 disabled
  server SRV_20 127.0.0.1:1 disabled
  server SRV_21 127.0.0.1:1 disabled
  server SRV_22 127.0.0.1:1 disabled
  server SRV_23 127.0.0.1:1 disabled
  server SRV_24 127.0.0.1:1 disabled
  server SRV_25 127.0.0.1:1 disabled
  server SRV_26 127.0.0.1:1 disabled
  server SRV_27 127.0.0.1:1 disabled
  server SRV_28 127.0.0.1:1 disabled
  server SRV_29 127.0.0.1:1 disabled
  server SRV_30 127.0.0.1:1 disabled
  server SRV_31 127.0.0.1:1 disabled
  server SRV_32 127.0.0.1:1 disabled
  server SRV_33 127.0.0.1:1 disabled
  server SRV_34 127.0.0.1:1 disabled
  server SRV_35 127.0.0.1:1 disabled
  server SRV_36 127.0.0.1:1 disabled
  server SRV_37 127.0.0.1:1 disabled
  server SRV_38 127.0.0.1:1 disabled
  server SRV_39 127.0.0.1:1 disabled
  server SRV_40 127.0.0.1:1 disabled
  server SRV_41 127.0.0.1:1 disabled
  server SRV_42 127.0.0.1:1 disabled

backend haproxy_svc_default-local-service_http from haproxytech
  mode http
  balance roundrobin
  option forwardfor
  no option abortonclose
  default-server check
  server SRV_1 127.0.0.1:6061 enabled
  server SRV_2 127.0.0.1:1 disabled
  server SRV_3 127.0.0.1:1 disabled
  server SRV_4 127.0.0.1:1 disabled
  server SRV_5 127.0.0.1:1 disabled
  server SRV_6 127.0.0.1:1 disabled
  server SRV_7 127.0.0.1:1 disabled
  server SRV_8 127.0.0.1:1 disabled
  server SRV_9 127.0.0.1:1 disabled
  server SRV_10 127.0.0.1:1 disabled
  server SRV_11 127.0.0.1:1 disabled
  server SRV_12 127.0.0.1:1 disabled
  server SRV_13 127.0.0.1:1 disabled
  server SRV_14 127.0.0.1:1 disabled
  server SRV_15 127.0.0.1:1 disabled
  server SRV_16 127.0.0.1:1 disabled
  server SRV_17 127.0.0.1:1 disabled
  server SRV_18 127.0.0.1:1 disabled
  server SRV_19 127.0.0.1:1 disabled
  server SRV_20 127.0.0.1:1 disabled
  server SRV_21 127.0.0.1:1 disabled
  server SRV_22 127.0.0.1:1 disabled
  server SRV_23 127.0.0.1:1 disabled
  server SRV_24 127.0.0.1:1 disabled
  server SRV_25 127.0.0.1:1 disabled
  server SRV_26 127.0.0.1:1 disabled
  server SRV_27 127.0.0.1:1 disabled
  server SRV_28 127.0.0.1:1 disabled
  server SRV_29 127.0.0.1:1 disabled
  server SRV_30 127.0.0.1:1 disabled
  server SRV_31 127.0.0.1:1 disabled
  server SRV_32 127.0.0.1:1 disabled
  server SRV_33 127.0.0.1:1 disabled
  server SRV_34 127.0.0.1:1 disabled
  server SRV_35 127.0.0.1:1 disabled
  server SRV_36 127.0.0.1:1 disabled
  server SRV_37 127.0.0.1:1 disabled
  server SRV_38 127.0.0.1:1 disabled
  server SRV_39 127.0.0.1:1 disabled
  server SRV_40 127.0.0.1:1 disabled
  server SRV_41 127.0.0.1:1 disabled
  server SRV_42 127.0.0.1:1 disabled

backend ssl-backend from haproxytech
  mode tcp
  server https unix@/var/run/ssl-frontend.sock send-proxy-v2

[oktalz]

hi @alexraitz-ship-it
we will take a look a it