Skip to content

Epic: Integrations #7714

@robert3005

Description

@robert3005

Vortex needs to have integrations into languages and frameworks that people want to use it in. This might mean improving API integrations, adding additional APIs to vortex to service integrations, having natural language bindings that are hard to use incorrectly.

Status

Languages

  • C/C++
  • Java (scan vortex files with java bindings, produce arrow)
  • Python
  • Java Panama (full API based on C API)

Engines

  • DuckDB
  • DataFusion
  • Spark
  • Iceberg
  • Trino
  • Velox
  • Substrait
  • Arrow
  • Polars
  • Pytorch
  • MegatronLM

Goal

The goal here is to reach a reasonable level in each one of these. This is not to say that no work is left but we can recommend people use it without caveats. You can see that for the purpose of this issue I have marked DuckDB and DataFusion as done even though there's still potential improvements to be made, these should be tracked separately. For any of the languages/frameworks that we work on under this Epic we have to define what reasonable looks like.

Motivation

These frameworks and languages are where the users are and where our usage lives. In order for vortex to be successful we have to penetrate all of the target user niches.

Unresolved questions

Instead of using language specific frameworks we could use UniFFI to generate all the bindings. Initially UniFFI didn't see to produce that useful of a client library but we should revisit.

Metadata

Metadata

Labels

epicPublic roadmap umbrella for a major initiative, with work tracked in sub-issues.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions