[GLUTEN-11550][UT] Enable Python-related tests and disable flaky JobTagging suite for Spark 4.x#11908
Draft
baibaichen wants to merge 3 commits intoapache:mainfrom
Draft
[GLUTEN-11550][UT] Enable Python-related tests and disable flaky JobTagging suite for Spark 4.x#11908baibaichen wants to merge 3 commits intoapache:mainfrom
baibaichen wants to merge 3 commits intoapache:mainfrom
Conversation
baibaichen
force-pushed
the
fix/11550-python
branch
from
7a11b8c to
24ae735
Compare
baibaichen
changed the title
baibaichen
force-pushed
the
fix/11550-python
branch
3 times, most recently
from
e64b6b0 to
5773249
Compare
- ArrowEvalPythonExecSuite: remove testWithMaxSparkVersion("4.0") on 3 arrow_udf tests
- GlutenRowQueueSuite: enable (spark41-only, 7/7 passed)
- GlutenPythonUDFSuite: enable with testGluten for SPARK-48706 (class name pattern fix)
- GlutenPythonDataSourceSuite: enable (all tests cancel without Python, won't fail CI)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The slow jobs were missing Python/pyspark installation, causing GlutenPythonUDTFSuite to fail with RuntimeException during suite loading. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
baibaichen
force-pushed
the
fix/11550-python
branch
from
5773249 to
bb57d47
Compare
…shdown Gluten replaces FilterExec with FilterExecTransformer and BatchScanExec with BatchScanExecTransformer. Add testGluten matching the Gluten operator names. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
baibaichen
force-pushed
the
fix/11550-python
branch
from
bb57d47 to
8420705
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Enable Python-related test suites and fix issues for Spark 4.1. Also disable a flaky suite.
ArrowEvalPythonExecSuite — Remove
testWithMaxSparkVersion("4.0")restriction on 3arrow_udftests. The underlying issues have been resolved.GlutenRowQueueSuite — Enable (spark41-only, 7/7 passed).
GlutenPythonUDFSuite — Enable with
testGlutenoverride forSPARK-48706. The original test usesthis.getClass.getSimpleNameinExpectedContextpattern which returnsGlutenPythonUDFSuitebut the actualcallSiterecordsPythonUDFSuite. ThetestGlutenfixes the pattern.GlutenPythonDataSourceSuite — Enable. All 23 tests cancel via
assume(shouldTestPythonUDFs)when Python is not available (no CI failure).GlutenPythonUDTFSuite — Fix to properly extend
PythonUDTFSuite(was an empty class). Enable. Tests cancel without Python.GlutenSparkSessionJobTaggingAndCancellationSuite — Disable (flaky). The
testGlutenfor "Tags set from session are prefixed with session UUID" suffers fromInheritableThreadLocal+ ForkJoinPool timing issues causing intermittentNoSuchElementException: head of empty list. Reverted testGluten and disabled the entire suite until root cause is fully resolved.How was this patch tested?
Related issue: #11550