new feature: support key-based rate limit

[dentiny]

Feature Description

Hi team, in production usage we've found hot key impacting storage service (the classic hotspotting issue for distributed system). I'm wondering apart from server-side support, if it's possible to add hotkey rate limit at client side as well, so we could

• reduce IO request proactively before service overloads
• 429 Too Many Requests is usually considered retriable, which could worsen production environment further (see new feature: introduce retry budget to prevent retry storm #7376 as well)

Problem and Solution

I'm wondering if we could add key-based rate limit, one viable data structure could be count-min sketch.

Additional Context

No response

Are you willing to contribute to the development of this feature?

• Yes, I am willing to contribute to the development of this feature.

[Xuanwo]

I think this feature makes a lot sense. For "key-based", I think what you want is more like "prefix-based"?

[dentiny]

For "key-based", I think what you want is more like "prefix-based"?

I'm thinking of two things:

• For object storage specifically, we could apply rate limit on bucket / directory
  • It matches how we create an S3 builder (which could be created via bucket and root), but I haven't abstracted it out to match other storage backends
• Internally we're using key-based rate limit, and seems to be useful to protect against spiky loads
  • Key range / prefix based sounds good as well!