Skip to content

Adds Iterable<DataRowSchema>.toDataFrame() overload#1907

Draft
Jolanrensen wants to merge 2 commits into
masterfrom
DataRowSchema-Iterable-toDataFrame
Draft

Adds Iterable<DataRowSchema>.toDataFrame() overload#1907
Jolanrensen wants to merge 2 commits into
masterfrom
DataRowSchema-Iterable-toDataFrame

Conversation

@Jolanrensen

Copy link
Copy Markdown
Collaborator

Fixes #1880

Added Iterable.toDataFrame() overload for DataRowSchema types,
compiler plugin test in :samples, because the compiler plugin is live there.

This fixes the compiler plugin creating a new type immediately when turning a list of @DataSchema annotated classes into a dataframe.

@Jolanrensen Jolanrensen added Compiler plugin Anything related to the DataFrame Compiler Plugin API If it touches our API labels Jun 22, 2026
@Jolanrensen

Copy link
Copy Markdown
Collaborator Author
image As expected, this directly touches the kotlin-spark dev example :D (where this limitation was detected) I'll fix the example right away

@Jolanrensen

Jolanrensen commented Jun 23, 2026

Copy link
Copy Markdown
Collaborator Author

A workaround for #1908 could be to write the function like this:

@JvmName("dataRowSchemaIterableToDataFrame")
@Suppress("INCORRECT_LEFT_COMPONENT_OF_INTERSECTION")
public inline fun <reified T : DataRowSchema> Iterable<T>.toDataFrame(): DataFrame<T & Any> =
    @DisableInterpretation
    toDataFrame {
        properties()
    }

But notebooks don't like T & Any and it could better be solved from the compiler plugin side I believe.

@koperagen

koperagen commented Jul 2, 2026

Copy link
Copy Markdown
Collaborator

Let's probably not merge since this needs a workaround if flexible type appears. A few things need to be fixed on compiler plugin side to properly work with such types. Let's wait these fixes in stable release of Kotlin, postpone feature to 1.1

Even this is not ideal because it suppresses INCORRECT_LEFT_COMPONENT_OF_INTERSECTION, which is not guaranteed to work in the future

public inline fun <reified T : DataRowSchema> Iterable<T>.toDataFrame(): DataFrame<T & Any> =
    @DisableInterpretation
    toDataFrame {
        properties()
    }

@Jolanrensen
Jolanrensen marked this pull request as draft July 2, 2026 09:58
@Jolanrensen Jolanrensen added this to the 1.1 milestone Jul 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

API If it touches our API Compiler plugin Anything related to the DataFrame Compiler Plugin

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Avoid breaking existing code with Iterable<DataSchema>.toDataFrame() calls when enabling the plugin

3 participants