Skip to content

HIVE-29688: HiveRelDecorrelator does not support semi-joins with correlation variables only in LHS #6567

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
thomasrebele wants to merge 1 commit into
base: master
Choose a base branch
from
+84 −7
Draft

HIVE-29688: HiveRelDecorrelator does not support semi-joins with correlation variables only in LHS #6567

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 16 additions & 7 deletions ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1353,7 +1353,7 @@ public Frame decorrelateRel(Join rel) {
throw new UnsupportedOperationException("Correlated subqueries in outer join conditions not supported yet." +
" Join condition: " + rel.getCondition());
}
if (!rel.getJoinType().projectsRight()) {
if (JoinRelType.ANTI.equals(rel.getJoinType())) {
return decorrelateRel((RelNode) rel);
}
//
Expand All @@ -1374,6 +1374,11 @@ public Frame decorrelateRel(Join rel) {
return null;
}

if (rel.getJoinType().projectsRight() && !rightFrame.corDefOutputs.isEmpty()) {
// decorrelating semi-join with correlation variables in the RHS is not supported yet
return null;
}

final RelNode newJoin = HiveJoin.getJoin(rel.getCluster(), leftFrame.r, rightFrame.r,
decorrelateExpr(rel.getCondition()), rel.getJoinType());

Expand All @@ -1385,16 +1390,20 @@ public Frame decorrelateRel(Join rel) {
int newLeftFieldCount = leftFrame.r.getRowType().getFieldCount();

int oldRightFieldCount = oldRight.getRowType().getFieldCount();
assert rel.getRowType().getFieldCount()
== oldLeftFieldCount + oldRightFieldCount;

int expectedRowCount = oldLeftFieldCount + (rel.getJoinType().projectsRight() ? oldRightFieldCount : 0);
if (rel.getRowType().getFieldCount() != expectedRowCount)
throw new AssertionError();

// Left input positions are not changed.
mapOldToNewOutputs.putAll(leftFrame.oldToNewOutputs);

// Right input positions are shifted by newLeftFieldCount.
for (int i = 0; i < oldRightFieldCount; i++) {
mapOldToNewOutputs.put(i + oldLeftFieldCount,
rightFrame.oldToNewOutputs.get(i) + newLeftFieldCount);
if (rel.getJoinType().projectsRight()) {
// Right input positions are shifted by newLeftFieldCount.
for (int i = 0; i < oldRightFieldCount; i++) {
mapOldToNewOutputs.put(i + oldLeftFieldCount,
rightFrame.oldToNewOutputs.get(i) + newLeftFieldCount);
}
}

final SortedMap<CorDef, Integer> corDefOutputs =
Expand Down
17 changes: 17 additions & 0 deletions ql/src/test/queries/clientpositive/decorrelate-semi-join.q
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
DROP TABLE IF EXISTS `tab`;
CREATE EXTERNAL TABLE `tab`(
`f1` string,
`f2` string,
`f3` string,
`f4` string,
`f5` string,
`f6` string);

SELECT 1
FROM tab a
WHERE a.f4 IN ('1', '2')
AND EXISTS (
SELECT 1
FROM tab b
WHERE a.f6 = b.f1 AND b.f3 IN (SELECT 1)
);
51 changes: 51 additions & 0 deletions ql/src/test/results/clientpositive/llap/decorrelate-semi-join.q.out
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
PREHOOK: query: DROP TABLE IF EXISTS `tab`
PREHOOK: type: DROPTABLE
PREHOOK: Output: database:default
POSTHOOK: query: DROP TABLE IF EXISTS `tab`
POSTHOOK: type: DROPTABLE
POSTHOOK: Output: database:default
PREHOOK: query: CREATE EXTERNAL TABLE `tab`(
`f1` string,
`f2` string,
`f3` string,
`f4` string,
`f5` string,
`f6` string)
PREHOOK: type: CREATETABLE
PREHOOK: Output: database:default
PREHOOK: Output: default@tab
POSTHOOK: query: CREATE EXTERNAL TABLE `tab`(
`f1` string,
`f2` string,
`f3` string,
`f4` string,
`f5` string,
`f6` string)
POSTHOOK: type: CREATETABLE
POSTHOOK: Output: database:default
POSTHOOK: Output: default@tab
Warning: Shuffle Join MERGEJOIN[29][tables = [$hdt$_1, $hdt$_2]] in Stage 'Reducer 2' is a cross product
PREHOOK: query: SELECT 1
FROM tab a
WHERE a.f4 IN ('1', '2')
AND EXISTS (
SELECT 1
FROM tab b
WHERE a.f6 = b.f1 AND b.f3 IN (SELECT 1)
)
PREHOOK: type: QUERY
PREHOOK: Input: _dummy_database@_dummy_table
PREHOOK: Input: default@tab
#### A masked pattern was here ####
POSTHOOK: query: SELECT 1
FROM tab a
WHERE a.f4 IN ('1', '2')
AND EXISTS (
SELECT 1
FROM tab b
WHERE a.f6 = b.f1 AND b.f3 IN (SELECT 1)
)
POSTHOOK: type: QUERY
POSTHOOK: Input: _dummy_database@_dummy_table
POSTHOOK: Input: default@tab
#### A masked pattern was here ####
Loading