perf: Improve efficiency of embedded kill tasks

[kfaraz]

Description

The UnusedSegmentsKiller currenly has some limitations:

• Wasted cycles: The kill queue contains jobs for all intervals which have an unused segment, regardless of whether the unused segment is eligible for kill or not.
• Slow progress: For intervals with a large number of killable unused segments, a single embedded kill task is launched which would kill only 1000 segments in one go. In such cases, the UnusedSegmentsKiller would take very long to clear a backlog of killable unused segments.
• When the kill queue is rebuilt, heavy metadata queries are fired for each datasource in the DB, even if they don't have any unused segments, killable or otherwise.

Changes

• While rebuilding the kill queue, fire a single metadata query to retrieve the intervals containing killable unused segments of all datasources.
  • This avoids firing multiple queries, one for each datasource in the DB.
  • It also ensures that every embedded kill task actually kills atleast 1 unused segment.
• In the above query, scan upto a maximum of 200k eligible unused segments. This would also be the maximum number of segments killed when the queue has been fully processed.
  • This number is large enough to make meaningful progress in each cycle of the UnusedSegmentsKiller and small enough to keep the query time within ~5s (tried on MySQL 8 with 17M unused segments).
• Pass in the max segments to kill in each KillCandidate.
• Allow each embedded kill task to kill multiple batches of segments, upto a maximum of 10 (previously 1).

---

This PR has:

• been self-reviewed.
  • using the concurrency checklist (Remove this item if the PR doesn't have any relation to concurrency.)
• added documentation for new or modified features or behaviors.
• a release note entry in the PR description.
• added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
• added or updated version, license, or notice information in licenses.yaml
• added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
• added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
• added integration tests.
• been tested in a test Druid cluster.

[cryptoe]

+1
I think the changes make sense to me.
Thank you for working on this.

[cryptoe]

So we have basically pushed in the bufferPeriod inside the sql query so that we donot populate segments which would be ineligible.
I think this change alone would get us lot of improvements.
I am a bit worried about fetching all the data sources at once but I guess there is where the inner query limit comes into place.

[kfaraz]

Yes, the inner limit (currently 200k) takes care of that.

Fetching all the datasources in one go is actually better since in the current flow, we would fire separate queries for each datasource. When the number of datasources is large, this can be pretty redundant, especially since a lot of the datasources might not even have killable unused segments.

[gianm]

On my machine this test legitimately takes nearly 30s to run, which has two problems:

• The timeout = 30_000L is low enough that it would make the test flaky
• 30s is just generally a long time.

Is it possible to change the test to run faster while still being effective?

[kfaraz]

Added a config to control the maxSegmentsToKill.
Most clusters should work fine with the default value of 200k itself.

[FrankChen021]

Severity	Findings
P0	0
P1	2
P2	1
P3	0
Total	3

The capped global scan can starve cleanup behind locked intervals, embedded tasks can exceed their discovered deletion limits, and malformed interval rows can abort queue rebuilding.

Reviewed 8 of 8 changed files.

---

This is an automated review by Codex GPT-5.6-Sol

[FrankChen021]

[P1] Prevent locked rows from monopolizing the global scan

The new global LIMIT has no ordering, cursor, or rotation state. If the selected 200,000 rows belong to intervals whose exclusive locks cannot be acquired, all generated tasks are skipped, the rows remain, and the next rebuild can select the identical slice again. Eligible segments outside that slice—including unrelated datasources—can then be starved indefinitely, whereas the previous per-datasource discovery still allowed other datasources to progress. Candidate discovery needs a deterministic progress/fairness mechanism that can advance past repeatedly skipped intervals.

[kfaraz]

While this is possible, it is not very likely to happen.

• Firstly, intervals will not remain locked forever, unless there is a lot of late arriving data in streaming supervisors.
• Secondly, the search query need not pick up the same intervals in the next iteration.

Also, the eventual goal is to use concurrent locks in embedded kill tasks, thereby removing the need to skip locked intervals altogether. I will create a follow up PR for that change.

[kfaraz]

Items for follow up PR:

• Document the new config
• Use concurrent locks