Skip to content

fix: scale controller PDB maxUnavailable with replica count - #1003

Closed
razvan wants to merge 3 commits into
mainfrom
fix/controller-pdb-max-unavailable
Closed

fix: scale controller PDB maxUnavailable with replica count#1003
razvan wants to merge 3 commits into
mainfrom
fix/controller-pdb-max-unavailable

Conversation

@razvan

@razvan razvan commented Aug 13, 2026

Copy link
Copy Markdown
Member

Description

Change the maxUnavailable for KRaft controller's PodDisruptionBudget from 1 to
(N - 1) / 2 controllers to ensure quorum health.

Definition of Done Checklist

  • Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
  • Please make sure all these things are done and tick the boxes

Author

  • Changes are OpenShift compatible
  • CRD changes approved
  • CRD documentation for all fields, following the style guide.
  • Helm chart can be installed and deployed operator works
  • Integration tests passed (for non trivial changes)
  • Changes need to be "offline" compatible
  • Links to generated (nightly) docs added
  • Release note snippet added

Reviewer

  • Code contains useful comments
  • Code contains useful logging statements
  • (Integration-)Test cases added
  • Documentation added or updated. Follows the style guide.
  • Changelog updated
  • Cargo.toml only contains references to git tags (not specific commits or branches)

Acceptance

  • Feature Tracker has been updated
  • Proper release label has been added
  • Links to generated (nightly) docs added
  • Release note snippet added
  • Add type/deprecation label & add to the deprecation schedule
  • Add type/experimental label & add to the experimental features tracker

The KRaft controller PodDisruptionBudget previously hardcoded
maxUnavailable to 1 regardless of how many controller replicas were
configured. Compute it from the total controller replica count
instead, keeping a strict Raft quorum majority available: at most
(N - 1) / 2 controllers may be unavailable at once, clamped to at
least 1 so node draining isn't blocked.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@razvan razvan self-assigned this Aug 13, 2026
@razvan razvan moved this to Development: Waiting for Review in Stackable Engineering Aug 13, 2026
@sbernauer
sbernauer self-requested a review August 13, 2026 08:09
@sbernauer

Copy link
Copy Markdown
Member

I'm in a meeting, but could you please add a little bit of motivation for this change? This is marked as a bugfix, where there any problems you ran into? I need to read a little bit what exactly the controllers do to be able to form an opinion.

In any case the docs in https://docs.stackable.tech/home/stable/kafka/usage-guide/operations/pod-disruptions/ should be updated to reflect the change, you can have a look at https://docs.stackable.tech/home/stable/trino/usage-guide/operations/pod-disruptions/ as a template

@razvan

razvan commented Aug 13, 2026

Copy link
Copy Markdown
Member Author

Thanks. I updated the documentation.

@sbernauer sbernauer moved this from Development: Waiting for Review to Development: In Review in Stackable Engineering Aug 13, 2026
@sbernauer

Copy link
Copy Markdown
Member

I have to admit I'm a bit surprised by this change.

could you please add a little bit of motivation for this change? This is marked as a bugfix, where there any problems you ran into?

That would be the most interesting part to me.

With (N - 1) / 2 we take away any safety margin, we are straight on the edge, where any Pod crash can bring the quorum down. IMO we should not do that and have some error margin.

If I read our concepts page on PDBs https://docs.stackable.tech/home/stable/concepts/operations/pod_disruptions, to me the controllers clearly fall below Multiple replicas to increase availability, so we should go with a conservative 1 (as-is, we would still need to fix the TODO and the docs).

PS: I also asked claude and it would also go with 1 (even without looking at existing docs). Happy to share the prompt/response.

Allow only a single Broker to be offline at any given time, regardless of the number of replicas or `roleGroups`.
This is because we can not make any assumptions about topic replication factors.

== Kraft Controllers

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
== Kraft Controllers
== KRaft Controllers

@razvan

razvan commented Aug 13, 2026

Copy link
Copy Markdown
Member Author

You are right. Apparently I misunderstood how PDBs work. Closing and leaving everything as is.

@razvan razvan closed this Aug 13, 2026
@sbernauer sbernauer mentioned this pull request Aug 13, 2026
20 tasks
@sbernauer

Copy link
Copy Markdown
Member

That's totally fair! Thanks for your willingness to compromise, appreciated! :)
I raised #1004 for the docs followup, feel free to have a look

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants