Closed
Conversation
Adds a method to replace a single child of an array by index, building on the existing with_children infrastructure. This is needed by the upcoming iterative execution scheduler which replaces children one at a time as they are executed. Signed-off-by: Nicholas Gates <nick@nickgates.com> Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds the ExecutionStep enum (ExecuteChild, ColumnarizeChild, Done) that encodings will return from VTable::execute instead of ArrayRef. This is infrastructure for the upcoming iterative execution scheduler. Signed-off-by: Nicholas Gates <nick@nickgates.com> Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…duler Changes the VTable::execute signature to return ExecutionStep instead of ArrayRef, and replaces the Executable for Columnar implementation with an iterative work-stack scheduler. The ExecutionStep enum has three variants: - ExecuteChild(i): ask the scheduler to execute child i to columnar - ColumnarizeChild(i): same but skip cross-step optimization - Done(result): execution complete The new scheduler in Executable for Columnar uses an explicit stack instead of recursion, and runs reduce/reduce_parent rules between steps via the existing optimizer infrastructure. All encoding implementations are mechanically wrapped in ExecutionStep::Done(...) to preserve existing behavior. Individual encodings will be migrated to use ExecuteChild/ColumnarizeChild in follow-up PRs. Signed-off-by: Nicholas Gates <nick@nickgates.com> Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com> Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move the iterative execution scheduler into a general-purpose `execute_until<M: Matcher>` method on `dyn Array`. The scheduler terminates when the root array matches `M`, while each child can specify its own termination condition via a `DonePredicate` carried in `ExecutionStep::ExecuteChild`. `ExecutionStep` now provides constructor methods: - `execute_child::<M>(idx)` — request child execution until M matches - `done(result)` — signal completion Both `Executable for Columnar` and `Executable for Canonical` are simplified to thin wrappers over `execute_until` with `AnyColumnar` and `AnyCanonical` matchers respectively. Signed-off-by: Nick Gates <nick@vortex.dev> Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace recursive child.execute() calls with ExecutionStep returns in Slice, Filter, Masked, FoR, and ZigZag vtables. Each now checks if its child is already in the needed form (canonical/primitive/constant) and returns Done directly, or returns ExecuteChild(0)/ColumnarizeChild(0) to let the scheduler handle child execution iteratively. Also handles ConstantArray children explicitly to prevent infinite loops in the scheduler (since constants are already columnar and won't be re-executed). FoR decompress is split into try_fused_decompress and apply_reference for reuse without recursive execution. Signed-off-by: Nicholas Gates <nick@nickgates.com> Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Convert Dict, ALP-RD, DecimalByteParts, and Zstd VTable::execute implementations to return ExecutionStep instead of recursively calling execute on children. Each encoding checks if children are already in the expected form (Primitive/Canonical/Constant) before proceeding, returning ExecuteChild(n) to let the scheduler handle child execution. Signed-off-by: Nicholas Gates <nick@nickgates.com> Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
ZstdVTable and ZstdBuffersVTable no longer recursively call .execute() after decompression. Instead they return Done with the decompressed intermediate, letting the scheduler re-enter execution on the result naturally. Signed-off-by: Nicholas Gates <nick@nickgates.com> Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… 4d) Convert the last two VTable::execute implementations that had recursive execute calls. SharedVTable now checks if current array (cached or source) is already columnar, otherwise returns ExecuteChild(0). ZstdBuffersVTable decompresses and returns Done(inner_array), letting the scheduler handle further execution of the decompressed result. Signed-off-by: Nicholas Gates <nick@nickgates.com> Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com> Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com> Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…cute-iter-more # Conflicts: # encodings/sequence/src/array.rs # encodings/zstd/public-api.lock # encodings/zstd/src/array.rs # encodings/zstd/src/zstd_buffers.rs # vortex-array/public-api.lock # vortex-array/src/arrays/bool/vtable/mod.rs # vortex-array/src/arrays/constant/vtable/mod.rs # vortex-array/src/arrays/decimal/vtable/mod.rs # vortex-array/src/arrays/extension/vtable/mod.rs # vortex-array/src/arrays/fixed_size_list/vtable/mod.rs # vortex-array/src/arrays/listview/vtable/mod.rs # vortex-array/src/arrays/null/mod.rs # vortex-array/src/arrays/primitive/vtable/mod.rs # vortex-array/src/arrays/struct_/vtable/mod.rs # vortex-array/src/arrays/varbinview/vtable/mod.rs # vortex-array/src/canonical.rs # vortex-array/src/executor.rs # vortex-array/src/vtable/dyn_.rs # vortex-array/src/vtable/mod.rs
…cute-iter-more # Conflicts: # vortex-array/public-api.lock # vortex-array/src/arrays/dict/vtable/mod.rs # vortex-array/src/arrays/filter/vtable.rs # vortex-array/src/arrays/masked/vtable/mod.rs # vortex-array/src/arrays/shared/vtable.rs # vortex-array/src/arrays/slice/vtable.rs
Merging this PR will degrade performance by 49.88%
Performance Changes
Comparing Footnotes
|
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes: #000
Testing