Add `replace { }.by { }` API for multi-column replacement using AddDsl

[Jolanrensen]

Converting our "movies" example to the compiler plugin, I came across the following notation:

.split { title }.by {
    listOf<Any>(
        """\s*\(\d{4}\)\s*$""".toRegex().replace(title, ""),
        "\\d{4}".toRegex().findAll(title).lastOrNull()?.value?.toIntOrNull() ?: -1,
    )
}.into("title", "year")

This creates the columns title: DataColumn<String> and year: DataColumn<Int>. The problem with this list notation is that the compiler plugin cannot read the types easily. It can only see them as Any, because that's the type of the list.

I tried several alternatives, but they all boiled down to multiple steps:

• add title2 and year, remove the old title, rename title2 to title
• replace or convert the title column to a column group of title/year, ungroup it again
• use split and then cast or requireColumn

But all seem to be way too complicated for what we want here: simply replacing a column with two new ones.

A new API could look like:

df.replace { title }.by { // AddDsl
    "title" from {
        """\s*\(\d{4}\)\s*$""".toRegex().replace(title, "")
    }
    "year" from {
        "\\d{4}".toRegex().findAll(title).lastOrNull()?.value?.toIntOrNull() ?: -1
    }
}

or more generally:

df.replace { name and firstName and age }.by {
    "name" from { 
         "$firstName $name ($age)"
    }
    "welcomeMessage" from {
        "Hi, $firstName!"
    }
}

We could explore other notations or names, of course, but it would narrow down to:

• remove some columns
• add some new ones

in one operation

[koperagen]

Personally, i'd like replace to map 1 column to 1 column group. For situations where i'd like to have a similar structure for 2 input columns
I like proposed use case of replacing multiple columns with multiple columns
Maybe we could extend merge?

merge { name and firstName and age }.into { 
    "name" from { 
         "$firstName $name ($age)"
    }
    "welcomeMessage" from {
        "Hi, $firstName!"
    }
}

I'd even be ok with replace by, but we'd still need some 1 column -> 1 column group mapping operation

[Jolanrensen]

merge is best suited for n -> 1 scenarios, I believe.

replace with for n -> n and replace by for n -> m could work, though we would need to be clear in the docs or come up with a clearer name.

[koperagen]

replace with for n -> n

problem is that it doesn't have AddDsl

[Jolanrensen]

Small alternative, similar to how we can combine groupBy and pivot, what about:

df.replace { name and firstName and age }.add/adding {
    "name" from { 
         "$firstName $name ($age)"
    }
    "welcomeMessage" from {
        "Hi, $firstName!"
    }
}

This reads like a sentence too "replace name, firstname, and age (by/while) adding 'name' from this and 'welcomeMessage' from that."

[koperagen]

1. Reserve replace operation for your proposed case with N -> M columns. replace with is relatively low level and we recommend convert asColumn instead, so new replace by can "pivot" :)) the overall meaning
2. Other API can be convert { col }.toColumnGroup { }?

[Jolanrensen]

and we recommend convert asColumn instead

except it cannot change names, unlike our current replace {}.with {}, however, we could even recommend replacing n -> n replace with with the n -> m one:

replace { colA }.with { 
    it.map { something(it) } named "other"
}

--->

replace { colA }.by { 
    colA.map { something(it) } into "other"
}

though it might be a bit more difficult to replace something like

replace { colA and colB }.with { it.named(it.name + "foo") }

but not impossible, we would just need to supply the selected columns in some way:

replace { colA and colB }.by { removedCols ->
    removedCols.forEach {
        it into (it.name + "foo")
    }
}

Other API can be convert { col }.toColumnGroup { }?

yeah sure :) or merge, or even convert asColumn where you create a column group. All works :)

[koperagen]

yeah sure :) or merge, or even convert asColumn where you create a column group. All works :)

I mean "add new API df.convert { }.toColumnGroup { CreateDataFrameDsl here }"

except it cannot change names, unlike our current replace {}.with {}, however, we could even recommend replacing n -> n replace with with the n -> m one:

replace { colA }.with {
it.map { something(it) } named "other"
}

I wouldn't recommend that, i think replace with is perfect for its purpose. What i meant to say is that: "i support the change in the meaning of operation because IMO replace by has a potential to be more popular than current replace with, which should still exist for original use case"