Skip to content

Commit 005f2a3

Browse files
ueshinHyukjinKwon
authored andcommitted
[SPARK-56168][PS][TESTS] Relax groupby diff-length assertions for pandas 3
### What changes were proposed in this pull request? This PR updates the pandas-on-Spark diff-frame groupby length test in `test_groupby_diff_len.py`. The changes are test-only: - restructure the test with `subTest` to make the failing case easier to identify - use `almost=True` for pandas 3.x - add a short comment explaining that pandas 3 includes external group keys for `as_index=False` and can widen their dtype after aligning mismatched lengths ### Why are the changes needed? The test behavior differs across pandas versions for mismatched-length external group keys. With pandas 3.x, `groupby(..., as_index=False)` can include the external group key in the result and widen its dtype after alignment. That makes the strict equality used by this test fail due to dtype differences even though the grouped values still match. This patch keeps the existing pandas 2 behavior untouched and relaxes the assertions only for pandas 3.x in this localized test. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Updated the related tests. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Codex (GPT-5) Closes #54969 from ueshin/issuse/SPARK-56168/diff_len. Authored-by: Takuya Ueshin <ueshin@databricks.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
1 parent 75757e1 commit 005f2a3

1 file changed

Lines changed: 36 additions & 31 deletions

File tree

python/pyspark/pandas/tests/diff_frames_ops/test_groupby_diff_len.py

Lines changed: 36 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@
1818
import pandas as pd
1919

2020
from pyspark import pandas as ps
21+
from pyspark.loose_version import LooseVersion
2122
from pyspark.pandas.config import set_option, reset_option
2223
from pyspark.testing.pandasutils import PandasOnSparkTestCase
2324
from pyspark.testing.sqlutils import SQLTestUtils
@@ -46,37 +47,41 @@ def test_groupby_different_lengths(self):
4647
pd.DataFrame({"a": [1, 2, 6, 4, 4, 6, 4, 3, 7], "b": [4, 2, 7, 3, 3, 1, 1, 1, 2]}),
4748
]
4849

49-
for pdf1, pdf2 in zip(pdfs1, pdfs2):
50-
psdf1 = ps.from_pandas(pdf1)
51-
psdf2 = ps.from_pandas(pdf2)
52-
53-
for as_index in [True, False]:
54-
if as_index:
55-
56-
def sort(df):
57-
return df.sort_index()
58-
59-
else:
60-
61-
def sort(df):
62-
return df.sort_values("c").reset_index(drop=True)
63-
64-
self.assert_eq(
65-
sort(psdf1.groupby(psdf2.a, as_index=as_index).sum()),
66-
sort(pdf1.groupby(pdf2.a, as_index=as_index).sum()),
67-
almost=as_index,
68-
)
69-
70-
self.assert_eq(
71-
sort(psdf1.groupby(psdf2.a, as_index=as_index).c.sum()),
72-
sort(pdf1.groupby(pdf2.a, as_index=as_index).c.sum()),
73-
almost=as_index,
74-
)
75-
self.assert_eq(
76-
sort(psdf1.groupby(psdf2.a, as_index=as_index)["c"].sum()),
77-
sort(pdf1.groupby(pdf2.a, as_index=as_index)["c"].sum()),
78-
almost=as_index,
79-
)
50+
for as_index in [True, False]:
51+
# pandas 3 includes external group keys for as_index=False and can widen
52+
# their dtype after aligning mismatched lengths.
53+
almost = as_index or LooseVersion(pd.__version__) >= "3.0.0"
54+
55+
if as_index:
56+
57+
def sort(df):
58+
return df.sort_index()
59+
60+
else:
61+
62+
def sort(df):
63+
return df.sort_values("c").reset_index(drop=True)
64+
65+
for i, (pdf1, pdf2) in enumerate(zip(pdfs1, pdfs2)):
66+
psdf1 = ps.from_pandas(pdf1)
67+
psdf2 = ps.from_pandas(pdf2)
68+
69+
with self.subTest(i=i, as_index=as_index):
70+
self.assert_eq(
71+
sort(psdf1.groupby(psdf2.a, as_index=as_index).sum()),
72+
sort(pdf1.groupby(pdf2.a, as_index=as_index).sum()),
73+
almost=almost,
74+
)
75+
self.assert_eq(
76+
sort(psdf1.groupby(psdf2.a, as_index=as_index).c.sum()),
77+
sort(pdf1.groupby(pdf2.a, as_index=as_index).c.sum()),
78+
almost=almost,
79+
)
80+
self.assert_eq(
81+
sort(psdf1.groupby(psdf2.a, as_index=as_index)["c"].sum()),
82+
sort(pdf1.groupby(pdf2.a, as_index=as_index)["c"].sum()),
83+
almost=almost,
84+
)
8085

8186

8287
class GroupByDiffLenTests(

0 commit comments

Comments
 (0)