feat(ingest): add max_shards cap to IngestSettings and ScalingArbiter#6470
Draft
fulmicoton-dd wants to merge 1 commit into
Draft
feat(ingest): add max_shards cap to IngestSettings and ScalingArbiter#6470fulmicoton-dd wants to merge 1 commit into
fulmicoton-dd wants to merge 1 commit into
Conversation
Adds `max_shards: Option<NonZeroUsize>` to `IngestSettings` so operators can bound the number of shards the control plane will open for a source. The scaling arbiter now accepts a `RangeInclusive<NonZeroUsize>` and clamps scale-up decisions within [min_shards, max_shards]; the scale-down path is unblocked when shards exceed the cap (e.g. after a config change).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
max_shards: Option<NonZeroUsize>toIngestSettings(defaults toNone= uncapped; backwards-compatible)max_shards >= min_shardsat index config build timeScalingArbiter::should_scalenow acceptsRangeInclusive<NonZeroUsize>instead of a baremin_shards; scale-up is capped atmax_shardsusingclampnum_open_shards >= max_shards, the scale-up block is skipped rather than returning early, so the scale-down path can still fire (e.g. when the cap is lowered after shards are already open)Test plan
cargo nextest run -p quickwit-control-plane -p quickwit-config --all-featurespassestest_scaling_arbiter_max_shardscovers: at-cap suppression, below-cap scale-up clamping, above-cap scale-down fall-throughingest_settings.max_shardsin an index config is respected at runtime