fix: clustered segment catalog spec complex handling - #19842
fix: clustered segment catalog spec complex handling#19842clintropolis wants to merge 2 commits into
Conversation
github-actions
Bot
added
Area - Batch Ingestion
Area - Segment Format and Ser/De
Area - MSQ
FrankChen021
left a comment
FrankChen021
left a comment
There was a problem hiding this comment.
| Severity | Findings |
|---|---|
| P0 | 0 |
| P1 | 0 |
| P2 | 1 |
| P3 | 0 |
| Total | 1 |
Reviewed 5 of 5 changed files.
This is an automated review by Codex GPT-5.6-Sol
| // Other complex types cannot be ingested into a clustered base table: there is no dimension handler for them, | ||
| // and clustered base tables have no aggregators to produce them. | ||
| if (druidType.is(ValueType.COMPLEX)) { | ||
| return complexDimensionSchema(column.name(), druidType); |
There was a problem hiding this comment.
[P2] Reject handler schemas that change the declared type
This path accepts the provider's DimensionSchema without checking that schema.getColumnType() equals the catalog's declared complex type. The new test demonstrates the problem by declaring COMPLEX but producing a DoubleDimensionSchema. During ingestion, DimensionSchema.getDimensionHandler() then selects the double handler from that returned schema, so complex values fail conversion or are stored as a type that contradicts the sealed catalog schema. Apply the same type-consistency check used for custom schemas before accepting the provider result.
2 participants
Description
ClusteredValueGroupsBaseTableMetadatanow checks for a complexDimensionHandlerfrom a new methodDimensionHandlerUtils#getComplexDimensionSchemawhich checksDIMENSION_HANDLER_PROVIDERSdirectly (instead of using the existingDimensionHandlerUtils#getHandlerFromCapabilitieswhich now shares the underlying logic with this new method).DimensionSchemaUtilshas also been updated to use this new method as it was already only callinggetHandlerFromCapabilitiesfor COMPLEX types.