feat: support partial loads for tasks

[clintropolis]

Description

This PR enables tasks to set SegmentLoaderConfig#virtualStoragePartialDownloadsEnabled to false (default true) via runtime property druid.indexer.task.virtualStoragePartialDownloadsEnabled or setting virtualStoragePartialDownloadsEnabled on MSQ query context.

[gianm]

IMO this should default to true, because it already requires v10 segments, which are opt-in. If you opt-in to v10 segments then you should get all the cool v10 stuff automatically.

[FrankChen021]

Severity	Findings
P0	0
P1	0
P2	2
P3	0
Total	2

Reviewed 19 of 19 changed files.

---

This is an automated review by Codex GPT-5.6-Sol

[FrankChen021]

[P2] Parse the query-context boolean before unboxing

This key is also exposed as an MSQ query-context parameter, whose QueryContext.getBoolean contract accepts string booleans. If a request supplies "false", that value is copied into the worker task context, but this raw generic lookup returns a String and unboxing it as boolean throws ClassCastException while constructing the toolbox, before IndexerWorkerContext can use the proper parser. Parse the value with QueryContext or the existing boolean conversion utility.

[FrankChen021]

[P2] Preserve the configured partial-download tuning

This fresh SegmentLoaderConfig only copies the enable flag, so per-task caches silently discard the configured metadata reservation and range-fetch/coalescing limits from the injected druid.segmentCache configuration. In particular, a V10 header larger than the hard-coded 16 MiB default still fails while directing the operator to raise druid.segmentCache.virtualStorageMetadataReservationEstimate, because that setting can never reach this cache. Derive this per-task config from the injected loader config before replacing the location and ephemeral flags.

[FrankChen021]

Severity	Findings
P0	0
P1	0
P2	1
P3	0
Total	1

The prior string-boolean and tuning-loss findings are addressed; this review covers the updated head.

Reviewed 21 of 21 changed files.

---

This is an automated review by Codex GPT-5.6-Sol

[FrankChen021]

[P2] Preserve only compatible virtual-storage settings

Copying the entire injected SegmentLoaderConfig also inherits numThreadsToLoadSegmentsIntoPageCacheOnDownload and numThreadsToLoadSegmentsIntoPageCacheOnBootstrap. If either valid operator setting is nonzero, every virtualStorage=true task cache now throws during SegmentLocalCacheManager construction because virtual storage rejects page-cache loading threads. Previously task caches started from defaults. Copy only the intended virtual-storage tuning fields or explicitly reset the incompatible settings.

[FrankChen021]

Severity	Findings
P0	0
P1	0
P2	1
P3	0
Total	1

Severity	Findings
P0	0
P1	0
P2	1
P3	0
Total	1

The prior page-cache-thread inheritance issue is fixed, but the updated setter introduces a configuration-deserialization risk.

Reviewed 22 of 22 changed files.

---

This is an automated review by Codex GPT-5.6-Sol

[FrankChen021]

[P2] Avoid sanitizing operator config through the Jackson setter

setVirtualStorage is also Jackson's bean mutator for the virtualStorage property, so JsonConfigProvider calls this branch while loading druid.segmentCache. Consequently operator config is not left untouched as promised: for example, virtualStorage=true with lazyLoadOnStart=true can have the latter applied first and then silently reset to false, changing bootstrap behavior on existing virtual-storage Historicals. Direct JSON also becomes property-order dependent. Keep this sanitization in withVirtualStorage(true) or a dedicated per-task derivation method, leave the bean setter as a simple assignment, and add a deserialization regression test.

[FrankChen021]

Severity	Findings
P0	0
P1	0
P2	2
P3	0
Total	2

The immutable creator/builder makes deserialization atomic, so the prior property-order finding is addressed. Two lifecycle and resource risks remain.

Reviewed 42 of 42 changed files.

Validation: Full and incremental git diff --check passed.

Findings that could not be attached inline:

• server/src/main/java/org/apache/druid/guice/StorageNodeModule.java:143 - [P2] Register the shared loading pool with the lifecycle. This qualified pool is scoped only as @LazySingleton, unlike the managed pool immediately above. Therefore its @LifecycleStop method is never registered with Druid's lifecycle. Once a virtual task resolves the provider, lifecycle shutdown does not cancel its executor or outstanding deep-storage reads, leaking the pool in embedded or restarted lifecycles. Use @ManageLifecycle, which already provides singleton lifecycle scope, instead.

---

This is an automated review by Codex GPT-5.6-Sol

[FrankChen021]

[P2] Keep node cache settings out of nonvirtual input caches

When virtualStorage is false, the production caller is DruidInputSource, but this branch now clones the process-wide druid.segmentCache configuration instead of the fresh defaults used before this PR. With numThreadsToLoadSegmentsIntoPageCacheOnDownload > 0, every reader creates and starts its own fixed executor; the input entity only drops segments and never calls shutdown(), so a persistent Indexer leaks a pool per reader/task. Inheriting deleteOnRemove=false also leaves downloaded input segments behind. Derive this branch from defaults or reset these transient cache settings; only virtual managers need the injected tuning.