Skip to content

feat: better validation for catalog types, more permissive parsing for native type strings - #19769

Open
clintropolis wants to merge 3 commits into
apache:masterfrom
clintropolis:tighter-catalog-spec-type-validation
Open

feat: better validation for catalog types, more permissive parsing for native type strings#19769
clintropolis wants to merge 3 commits into
apache:masterfrom
clintropolis:tighter-catalog-spec-type-validation

Conversation

@clintropolis

Copy link
Copy Markdown
Member

Description

This PR improves catalog type validation to reject invalid complex types instead of silently being ignored. It also updates druid native type validation to perform case insensitive handling of ARRAY and COMPLEX type prefixes, to be more consistent with how primitive types are handled (case insensitive), and fixes bug with handling of malformed type strings without a closing >. Prior to this something like long was allowed, but complex<json> or array<long> was not allowed (while ARRAY<long> was since array parsing recurses after removing the prefix). Array and complex prefix handling is now case insensitive, but complex type name handling itself must still be exact, as the underlying complex type registry is still case sensitive.

@FrankChen021 FrankChen021 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity Findings
P0 0
P1 1
P2 0
P3 0
Total 1
Severity Findings
P0 0
P1 1
P2 0
P3 0
Total 1

Reviewed 6 of 6 changed files.


This is an automated review by Codex GPT-5.6-Sol

return typeFactory.ofArray(elementType);
}
if (typeString.startsWith(COMPLEX_PREFIX)) {
if (upperTypeString.startsWith(COMPLEX_PREFIX) && upperTypeString.endsWith(">")) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Reject malformed declared types instead of returning null

The new closing-bracket guards make declarations such as COMPLEX<json and ARRAY<LONG fall through to null. Catalog consumers treat null as an undeclared type and default it to STRING, so a malformed schema can pass validation and ingest with the wrong physical type. Throw or validate declared types before this fallback.

@FrankChen021 FrankChen021 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity Findings
P0 0
P1 1
P2 1
P3 0
Total 2

The previous missing-closing-bracket finding is addressed across the parser, SQL EXTEND, clustered base tables, and datasource catalogs. External catalog tables still silently coerce malformed declarations to STRING, and invalid array element types can escape as a 500.

Reviewed 13 of 13 changed files.


This is an automated review by Codex GPT-5.6-Sol

// validate, so tables stored before this rule keep resolving (though editing them surfaces the invalid type).
// Columns.druidType maps __time to LONG regardless of the declared type, which ColumnSpec.validate already
// restricts to LONG or untyped.
if (spec.dataType() != null && Columns.druidType(spec) == null) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Validate declared types for external catalog tables

The new type check is scoped to DatasourceDefn, while ExternalTableDefn.validateColumn remains empty. Posting an external catalog table with dataType COMPLEX<json (now parsed as null by this PR) or FOO therefore succeeds; BaseInputSourceDefn later calls Columns.convertSignature, which replaces the null type with STRING. This preserves the silent schema substitution this PR intends to reject and can make external ingestion use a different type than declared. Move this parse check into shared table/column validation or add equivalent external-table validation.

// taken from the original string: complex type names are case-sensitive registry keys which must be preserved
// in their original casing (array element types recurse through this method, so any casing works for them).
// A parameterized type without the closing bracket is malformed, not a truncated parameter name.
if (upperTypeString.startsWith(ARRAY_PREFIX) && upperTypeString.endsWith(">")) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Handle unknown array element types as invalid input

ARRAY<FOO> passes this prefix/bracket guard, but the recursive parse returns null and Preconditions.checkNotNull throws NullPointerException. DatasourceDefn now passes every declared catalog type through this parser, while CatalogResource maps only IAE and DruidException to a 400 response, so a datasource without base-table metadata submitted with this invalid array type produces a 500 instead of the intended invalid-input response. Return null here or convert this case to InvalidInput, and add coverage for unsupported array element types.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants