Tracking Issue: Extension Arrays

[gatesn]

Motivation

Extension Arrays x ScalarFns is broken right now and would require a second level of dispatch via the ExtensionArray vtable.

Design

ExtensionArray::id() returns the ExtDType::id(), same as ScalarFnArray.

ArrayVTable::execute_parent() disappears in favor of a session-based registry.

Extension x ScalarFn can now be defined by either the extension author, or the scalar function author, or a third-party and dispatch always works in the same way.

Structural validation is performed in try_new. Value validation is a separate constructor that takes an execution context.

impl ExtensionArray {
    pub fn try_new(ext_dtype: ExtDTypeRef, storage: ArrayRef) -> VortexResult<Self>;

    pub fn try_new_validated(
        ext_dtype: ExtDTypeRef,
        storage: ArrayRef,
        ctx: &mut ExecutionCtx,
    ) -> VortexResult<Self>;

    pub unsafe fn new_unchecked(ext_dtype: ExtDTypeRef, storage: ArrayRef) -> Self;
}

Steps

• Move execute_parent to session
• Add ArrayPlugin to deserialize vortex.ext into an ExtensionArray with the correct ID
• Change ExtensionArray to return ExtDType::id

Unresolved questions

• None yet.

Implementation history

[joseph-isaacs]

How do we check if an extension array is canonical? I guess a method on the ExtVTable?