feat: better validation for catalog types, more permissive parsing for native type strings - #19769
feat: better validation for catalog types, more permissive parsing for native type strings#19769clintropolis wants to merge 3 commits into
Conversation
…r native type strings
FrankChen021
left a comment
There was a problem hiding this comment.
| Severity | Findings |
|---|---|
| P0 | 0 |
| P1 | 1 |
| P2 | 0 |
| P3 | 0 |
| Total | 1 |
| Severity | Findings |
|---|---|
| P0 | 0 |
| P1 | 1 |
| P2 | 0 |
| P3 | 0 |
| Total | 1 |
Reviewed 6 of 6 changed files.
This is an automated review by Codex GPT-5.6-Sol
| return typeFactory.ofArray(elementType); | ||
| } | ||
| if (typeString.startsWith(COMPLEX_PREFIX)) { | ||
| if (upperTypeString.startsWith(COMPLEX_PREFIX) && upperTypeString.endsWith(">")) { |
There was a problem hiding this comment.
[P1] Reject malformed declared types instead of returning null
The new closing-bracket guards make declarations such as COMPLEX<json and ARRAY<LONG fall through to null. Catalog consumers treat null as an undeclared type and default it to STRING, so a malformed schema can pass validation and ingest with the wrong physical type. Throw or validate declared types before this fallback.
FrankChen021
left a comment
There was a problem hiding this comment.
| Severity | Findings |
|---|---|
| P0 | 0 |
| P1 | 1 |
| P2 | 1 |
| P3 | 0 |
| Total | 2 |
The previous missing-closing-bracket finding is addressed across the parser, SQL EXTEND, clustered base tables, and datasource catalogs. External catalog tables still silently coerce malformed declarations to STRING, and invalid array element types can escape as a 500.
Reviewed 13 of 13 changed files.
This is an automated review by Codex GPT-5.6-Sol
| // validate, so tables stored before this rule keep resolving (though editing them surfaces the invalid type). | ||
| // Columns.druidType maps __time to LONG regardless of the declared type, which ColumnSpec.validate already | ||
| // restricts to LONG or untyped. | ||
| if (spec.dataType() != null && Columns.druidType(spec) == null) { |
There was a problem hiding this comment.
[P1] Validate declared types for external catalog tables
The new type check is scoped to DatasourceDefn, while ExternalTableDefn.validateColumn remains empty. Posting an external catalog table with dataType COMPLEX<json (now parsed as null by this PR) or FOO therefore succeeds; BaseInputSourceDefn later calls Columns.convertSignature, which replaces the null type with STRING. This preserves the silent schema substitution this PR intends to reject and can make external ingestion use a different type than declared. Move this parse check into shared table/column validation or add equivalent external-table validation.
| // taken from the original string: complex type names are case-sensitive registry keys which must be preserved | ||
| // in their original casing (array element types recurse through this method, so any casing works for them). | ||
| // A parameterized type without the closing bracket is malformed, not a truncated parameter name. | ||
| if (upperTypeString.startsWith(ARRAY_PREFIX) && upperTypeString.endsWith(">")) { |
There was a problem hiding this comment.
[P2] Handle unknown array element types as invalid input
ARRAY<FOO> passes this prefix/bracket guard, but the recursive parse returns null and Preconditions.checkNotNull throws NullPointerException. DatasourceDefn now passes every declared catalog type through this parser, while CatalogResource maps only IAE and DruidException to a 400 response, so a datasource without base-table metadata submitted with this invalid array type produces a 500 instead of the intended invalid-input response. Return null here or convert this case to InvalidInput, and add coverage for unsupported array element types.
Description
This PR improves catalog type validation to reject invalid complex types instead of silently being ignored. It also updates druid native type validation to perform case insensitive handling of
ARRAYandCOMPLEXtype prefixes, to be more consistent with how primitive types are handled (case insensitive), and fixes bug with handling of malformed type strings without a closing>. Prior to this something likelongwas allowed, butcomplex<json>orarray<long>was not allowed (whileARRAY<long>was since array parsing recurses after removing the prefix). Array and complex prefix handling is now case insensitive, but complex type name handling itself must still be exact, as the underlying complex type registry is still case sensitive.