Skip to content

Extend datafusion-spark sequence function #22008

@erenavsarogullari

Description

@erenavsarogullari

Is your feature request related to a problem or challenge?

This is follow-up issue for datafusion-spark sequence function: #20955 (comment)

Following cases also need to be supported:
Currently, following existing test cases having Interval (step) definition for both TIMESTAMP and DATE work as expected:
https://github.com/apache/datafusion/pull/20955/changes#diff-f38c343ca9807a82465bea986e7bd1417947b26407dbd82e91ed328b6c15500dR134

However, the problem occurs when Interval (step) definition is not set as follows:
Spark:

spark.sql("SELECT sequence(TIMESTAMP'2023-01-03T00:00:00', TIMESTAMP'2023-01-01T00:00:00');")
[ArraySeq(2023-01-03 00:00:00.0, 2023-01-02 00:00:00.0, 2023-01-01 00:00:00.0)]

spark.sql("SELECT sequence(DATE'2018-01-03', DATE'2018-01-01');")
[ArraySeq(2018-01-03, 2018-01-02, 2018-01-01)]

DataFusion:

SELECT sequence(DATE '2018-01-04', DATE '2018-01-01');
actual: []
expected: [2018-01-04, 2018-01-03, 2018-01-02, 2018-01-01]

SELECT sequence(TIMESTAMP '2023-01-03T00:00:00', TIMESTAMP '2023-01-01T00:00:00');
actual: []
expected: [2023-01-03T00:00:00, 2023-01-02T00:00:00, 2023-01-01T00:00:00]

Describe the solution you'd like

Has been explained at the first section.

Describe alternatives you've considered

No response

Additional context

No response

Metadata

Metadata

Labels

enhancementNew feature or request
No fields configured for Feature.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions