pulsar-4.6.0-candidate-1

ZooKeeper and Broker Services split into ClusterIP + headless

Note: Upgrading existing installs may cause a brief service disruption. The StatefulSet's serviceName is immutable, so the ZooKeeper and Broker StatefulSets are re-created during the upgrade (see Upgrading below).

PRs #649 and #650 replace the single Service that fronted each of the ZooKeeper and Broker StatefulSets with two:

• a regular ClusterIP Service (<release>-zookeeper, <release>-broker) — used by clients; only routes to ready pods.
• a headless Service (*-headless, clusterIP: None, publishNotReadyAddresses: true) — used as the StatefulSet serviceName for stable per-pod DNS.

Why

ZooKeeper. The previous Service had publishNotReadyAddresses: true, so brokers and bookies could be routed to ZK pods that were still starting or unhealthy. Splitting into a ready-only ClusterIP Service for clients and a headless Service for per-pod DNS fixes that.

Brokers (issue #437). A broker registers itself in ZooKeeper using its per-pod DNS name; other brokers and clients then resolve that name to reach it. The previous headless Service did not set publishNotReadyAddresses, so the per-pod name only became resolvable after the pod's readiness probe passed (plus DNS-cache TTL). Meanwhile the load manager could already have assigned namespace bundles to the new broker, causing a brief disruption on those topics. The new headless Service sets publishNotReadyAddresses: true, so the per-pod name resolves immediately. Two further benefits:

• Client lookups now go through a regular ClusterIP Service that returns a single IP. The previous headless Service returned one A record per broker, which can exceed the 512-byte UDP DNS limit in larger clusters. Some DNS clients cannot handle this due to lack of TCP fallback for DNS (for example Alpine <3.18).
• StatefulSets require a headless Service for pod identity, so the headless Service can only be paired with — not replaced by — a ClusterIP Service.

Upgrading: pre-upgrade cleanup Job

Because serviceName is immutable, an in-place upgrade from a pre-4.6.0 chart would fail. The chart ships a pre-upgrade Job per component that uses kubectl (image images.kubectl, default alpine/k8s) to delete the old StatefulSet with --cascade=orphan. Pods (and ZK on-disk data) are preserved and keep running until the new StatefulSet rolls them, but a brief disruption around the cutover is possible. The Job reads the existing chart label and only acts when the prior version is < 4.6.0; disable with zookeeper.statefulsetUpgrade.enabled=false or broker.statefulsetUpgrade.enabled=false to manage the migration manually.

GitOps users (ArgoCD, Flux, Pulumi, etc.): the cleanup relies on Helm's pre-upgrade hook lifecycle, which isn't always honored by GitOps tooling that renders the chart and applies the manifests directly. Verify that your tool runs helm.sh/hook: pre-upgrade Jobs before the rest of the release — or disable the hook flags above and handle the StatefulSet deletion (with --cascade=orphan) as part of your migration — before upgrading to 4.6.0.

TLS

Certificates now include the new *-headless DNS names as SANs. After cert-manager reissues them, do a rolling restart of ZooKeeper and brokers so the running pods pick up matching certificates.

In-chart JWT secret generation

PR #672 (by @smbecker) removes the need to run prepare_helm_release.sh — or any out-of-band script — to seed JWT secrets before installing.

Opt in with auth.authentication.jwt.generateSecrets.enabled: true. A pre-install/pre-upgrade Job mints the signing key (symmetric or RSA) and one token per auth.superUsers entry, storing them as the same <release>-token-* secrets the rest of the chart already consumes. The Job is idempotent — skipped if the signing key secret exists, and existing token secrets are never overwritten — and supports annotations on generated secrets for tooling like reflector. Default is false, so existing installs are unaffected.

A fully-authenticated cluster can now be deployed with a single helm install.

Standalone deployment mode

PR #674 (by @smbecker) adds a top-level standalone toggle that deploys a single Pulsar standalone instance instead of separate ZooKeeper, BookKeeper, Broker, etc. workloads.

The goal is to use the same Helm chart for minimal development and test deployments on Kubernetes — local Kind/k3d/minikube, ephemeral CI, developer sandboxes — without a separate chart or installer. Existing values, image overrides, and tooling carry over.

What's Changed

• Add separate headless service for ZooKeeper statefulset by @lhotari in #649
• Add separate headless service for brokers with publishNotReadyAddresses by @lhotari in #650
• Revisit configurationMetadataStoreUrl and bookkeeperMetadataServiceUri settings by @lhotari in #652
• Fix pulsar_metadata.metadataStoreAllowReadOnlyOperations with ZooKeeper by @lhotari in #653
• Remove statsProviderClass config for bookkeeper by @lhotari in #654
• Enable local sessions in ZK when readonly mode is enabled by @lhotari in #655
• fix: fix extra volumes indentation for rendering the helm chart properly by @soyji in #656
• Fix CI failures with OpenID jobs by @lhotari in #659
• Upgrade VictoriaMetrics to a supported version by @lhotari in #657
• Upgrade to Oxia 0.16.0 by @lhotari in #658
• fix: remove usage of tty to avoid introducing \r by @soyji in #661
• Apply ShellCheck fixes to bash scripts by @frankgrimes97 in #663
• add support for custom annotations on initialization Jobs by @lyuval-armosec in #665
• fix: Add chart control for updateStrategy to brokers and proxies by @darinspivey in #668
• Add support for ipAddresses in TLS certificate templates by @ankith-carver in #643
• Enable TLS certificate file refresh for BookKeeper components by default by @lhotari in #669
• Add support for generating JWT secrets during install by @smbecker in #672
• Add cert annotation configuration by @smbecker in #673
• Add support for deploying pulsar as standalone by @smbecker in #674
• Fix oxia coordinator RoleBinding roleRef.apiGroup to prevent GitOps drift by @lhotari in #676
• Update standalone defaults to use zookeeper by @smbecker in #675
• fix: Improve Zookeeper initialization wait logic to support multi url configuration store by @ganeshkalyank in #671
• Upgrade appVersion to 4.0.10 so that Pulsar 4.0.10 image is used by default by @lhotari in #677
• Upgrade Oxia to 0.16.3 by @lhotari in #678

New Contributors

• @soyji made their first contribution in #656
• @frankgrimes97 made their first contribution in #663
• @lyuval-armosec made their first contribution in #665
• @darinspivey made their first contribution in #668
• @ankith-carver made their first contribution in #643
• @smbecker made their first contribution in #672
• @ganeshkalyank made their first contribution in #671

Full Changelog: pulsar-4.5.0...pulsar-4.6.0-candidate-1

pulsar-4.5.0

What's Changed

• Increase max heap size from 64m -> 128m for proxy and autorecovery by @lhotari in #642
• Fix ssh-access to CI in forks by @lhotari in #644
• Upgrade Oxia to 0.15.1 by @lhotari in #641
• Upgrade Dekaf UI to 1.1.0 by @visortelle in #646
• Add support for BookKeeper indexDirectories configuration by @Denovo1998 in #645
• Upgrade to Oxia 0.15.3 by @lhotari in #647
• Upgrade to Pulsar 4.0.9 by @lhotari in #648

New Contributors

• @Denovo1998 made their first contribution in #645

Full Changelog: pulsar-4.4.0...pulsar-4.5.0

pulsar-4.5.0-candidate-1

What's Changed

• Increase max heap size from 64m -> 128m for proxy and autorecovery by @lhotari in #642
• Fix ssh-access to CI in forks by @lhotari in #644
• Upgrade Oxia to 0.15.1 by @lhotari in #641
• Upgrade Dekaf UI to 1.1.0 by @visortelle in #646
• Add support for BookKeeper indexDirectories configuration by @Denovo1998 in #645
• Upgrade to Oxia 0.15.3 by @lhotari in #647
• Upgrade to Pulsar 4.0.9 by @lhotari in #648

New Contributors

• @Denovo1998 made their first contribution in #645

Full Changelog: pulsar-4.4.0...pulsar-4.5.0-candidate-1

pulsar-4.4.0

What's Changed

• Initial Dekaf UI support by @visortelle in #637
• Minor README improvements by @visortelle in #638
• Upgrade to Pulsar 4.0.8 by @lhotari in #640

New Contributors

• @visortelle made their first contribution in #637

Full Changelog: pulsar-4.3.0...pulsar-4.4.0

pulsar-4.4.0-candidate-1

What's Changed

• Initial Dekaf UI support by @visortelle in #637
• Minor README improvements by @visortelle in #638
• Upgrade to Pulsar 4.0.8 by @lhotari in #640

New Contributors

• @visortelle made their first contribution in #637

Full Changelog: pulsar-4.3.0...pulsar-4.4.0-candidate-1

pulsar-4.3.0

What's Changed

• Upgrade Oxia image to oxia/oxia:0.14.4 by @lhotari in #625
• Replace G1GC with generational ZGC and simplify PULSAR_GC options in values.yaml by @lhotari in #626
• Test with k8s v1.34.0 by @lhotari in #628
• Add support for ClusterIssuer for certificates by @lhotari in #630
• Fix tolerations for bookie init by @lhotari in #632
• Configure automatic compaction of the Bookie entry location index RocksDB database by @lhotari in #633
• Correct VM Operator CRDs version by @mnit016 in #634
• Upgrade to Pulsar 4.0.7 by @lhotari in #635

Full Changelog: pulsar-4.2.0...pulsar-4.3.0

pulsar-4.3.0-candidate-1

What's Changed

• Upgrade Oxia image to oxia/oxia:0.14.4 by @lhotari in #625
• Replace G1GC with generational ZGC and simplify PULSAR_GC options in values.yaml by @lhotari in #626
• Test with k8s v1.34.0 by @lhotari in #628
• Add support for ClusterIssuer for certificates by @lhotari in #630
• Fix tolerations for bookie init by @lhotari in #632
• Configure automatic compaction of the Bookie entry location index RocksDB database by @lhotari in #633
• Correct VM Operator CRDs version by @mnit016 in #634
• Upgrade to Pulsar 4.0.7 by @lhotari in #635

Full Changelog: pulsar-4.2.0...pulsar-4.3.0-candidate-1

pulsar-4.2.0

Upgrading from previous releases

TLS configuration for ZooKeeper has changed. Please see detailed instructions for upgrading.

What's Changed

• Use PEM files directly as ZooKeeper keystore and truststore by @lhotari in #613
• Add labels to all k8s objects by @lhotari in #617
• extra volume mounts for oxia coordinator by @anosulchik in #618
• Feat/cacerts by @gulecroc in #619
• extra volume mounts for autorecovery [init] containers by @anosulchik in #620
• Upgrade to Pulsar 4.0.6 by @lhotari in #622

Full Changelog: pulsar-4.1.0...pulsar-4.2.0

pulsar-4.2.0-candidate-1

Upgrading from previous releases

TLS configuration for ZooKeeper has changed. Please see detailed instructions for upgrading.

What's Changed

• Use PEM files directly as ZooKeeper keystore and truststore by @lhotari in #613
• Add labels to all k8s objects by @lhotari in #617
• extra volume mounts for oxia coordinator by @anosulchik in #618
• Feat/cacerts by @gulecroc in #619
• extra volume mounts for autorecovery [init] containers by @anosulchik in #620
• Upgrade to Pulsar 4.0.6 by @lhotari in #622

Full Changelog: pulsar-4.1.0...pulsar-4.2.0-candidate-1

pulsar-4.1.0

Upgrading to 4.1.0

This version introduces OpenID authentication. Setting auth.authentication.provider is no longer supported, you need to enable the provider with auth.authentication.<provider>.enabled.

What's Changed

• oxia components podmonitor match labels by @anosulchik in #605
• add support of extra volumes and mounts for autorecovery by @anosulchik in #607
• Add appAnnotations for all statefulsets by @HaimKortovich in #604
• update nodeSelector for bookkeeper cluster initializer by @bpoole6 in #608
• Oxia coordinator and server pod annotations by @anosulchik in #610
• configurable oxia coordinator configmap and entrypoint by @anosulchik in #606
• extra containers and volumes for oxia coordinator by @anosulchik in #609
• Upgrade oxia image to 0.12.0 in default values.yaml by @lhotari in #611
• OpenID: introducing support for OpenID configuration by @brunodomenici in #509
• Fix broker initialization error when using global Zookeeper (#602) by @trynocoding in #603
• Upgrade to Pulsar 4.0.5 by @lhotari in #612
• add template for ca issuer name and secret name by @gulecroc in #565

New Contributors

• @anosulchik made their first contribution in #605
• @HaimKortovich made their first contribution in #604
• @bpoole6 made their first contribution in #608
• @brunodomenici made their first contribution in #509
• @trynocoding made their first contribution in #603

Full Changelog: pulsar-4.0.1...pulsar-4.1.0