[Suggestion] Outstanding PRs by @Ayanda-D

[michaelklishin]

RabbitMQ series

4.2.x

Operating system (distribution) used

Linux

How is RabbitMQ deployed?

Generic binary package

What would you like to suggest for a future version of RabbitMQ?

Moved from #14305 to discuss the contributor opinion on a few PRs by @Ayanda-D:

• rabbitmq-queues: a way to grow a quorum queue member set to a target size #13873: merged ✅ as #13873 by @Ayanda-D, polished and rebased #15166
• Make it possible to shrink only a subset of quorum queues by providing a naming pattern #15021 (merged ✅)
• rabbitmq-queues: a command to display a member with highest (commit, log, snapshot) index #14237: reviewed by a fairly regular contributor and so far, seems to be only effective some of the time

[deadtrickster]

Regarding #13873, The very nature of the command gives me that bulk vibe. The concern is if it is intended do be some kind of "emergency" operation (but also if it is not, but a little bit less) the already stressed system will get tons of rpc calls and influx of new members - raft sync, snapshot sync, reelection cycles. Was this tested?

[Ayanda-D]

The very nature of the command gives me that bulk vibe

Not sure I fully get what you mean by "bulk vibe"? We take growing QQs as a once off operation (or rarely executed procedure) - this extension is using the existing "grow ", only difference being it picks the <node> randomly for the user per loop, and grows until <target-size> is reached. The grow operation itself is still the same. Currently, doing this one <node> per time from the CLI is very inefficient and requires node-name awareness (and results in multiple RPC calls from the rabbitmq-queues grow). We dont really mind which nodes the QQ grows to (as long they are healthy and available), all we want is target-quorum-cluster-size to be reached.

[Ayanda-D]

just a few follow up notes regarding these proposed new tools:

#13873 - CMQs had 3 HA-Policies for replicating a queue: all, exactly-n and nodes. The least used nodes policy was partly/mainly due to node-awareness of the policy. Using all, exactly-n (the most used policies from what we've seen) was easy & straight-forward. Single policy and the broker does all the work, without any node-name awareness to manage. Which is what we are trying to somewhat make available with QQs on this change. & as mentioned, nothing's changed much beneath - same grow operation, just added random node selection logic for the caller, on the broker (instead of the CLI). Current grow <node> using hardcoded nodenames is restrictive and slower (just doing grow_to_count N and let the broker do most of the processing work).

#15021 - this just helps in filtering/matching queues to be shrunk - when all queues are not required to force shrink on per vhost or globally.

#14237 - this is to help avoid parsing the quorum-status output to get the same info - and doing comparisons from external tools. Processing on the broker is faster and less data sent between the broker and cli-node (this is just for finer visibility on QQ highest member indices - no further implications on the queue). The quorum-status json output can be used to get the same information, but the exchanged payload between node & cli is big and more processing on the caller.

We don't see these updates lying on any critical path to affect performance - hence low risk? unless folks see otherwise (perhaps #13873 & #15021 to be carried out planned periods). These tools and extensions are just to assist on QQ operational procedures.

[michaelklishin]

I am trying to reach consensus on merging #13873 which seems to have received a more neutral review than the rabbitmq-queues shrink extension.

[michaelklishin]

✅ Merged as #13873.

Next up for our internal debate is #15021.

[michaelklishin]

#13873 is in with a few tweaks (that do not change the idea or CLI interface) #15166.

[michaelklishin]

#15021 is in.

[michaelklishin]

@Ayanda-D both PRs that I could, however slowly and painfully, reach consensus on with the rest of the core team, are now merged.

I hope this gives you folks enough space to implement the QQ recovery procedure you want to have as an option.

As for #14237, looks like it is not as useful as originally expected, according to reviews. But even if I'm wrong, #14237 looks like something a custom plugin can reimplement in a separate module.

[Ayanda-D]

@michaelklishin thank you! & yes, this goes a long way on having clear QQ recovery path/plan (those 2 are more critical, and #14237 for improved visibility).

Look forward to getting our hands on v4.2.3 🤗 !!