MDEV-39499 Updates to derived-with-keys, window functions determining records per key by mariadb-RexJohnston · Pull Request #5020 · MariaDB/server

mariadb-RexJohnston · 2026-04-29T19:06:37Z

Enabling derived keys optimization for derived.col = const pushed conditions.

Estimating records per key in derived key for the optimizer based on form and/or size of components of a derived table.

Consider a derived table of the form

SELECT ..., ROW_NUMBER () OVER (PARTITION BY c1,c2 order by ...) FROM t1, t2, t3 ...
WHERE ...

If the optimizer generates a key on this derived table because of a constraint being pushed into it, it currently will not consider key components of the form col = const

We lift this constraint and add code to TABLE::add_tmp_key to search for a window function ROW_NUMBER(). From the partition list c1, c2 we can in infer an estimate of the number of rows we expect to see for each key value. The optimizer can then use this number to determine a better table join order.

spetrunia · 2026-05-03T18:04:53Z

I was just looking at the patch, mrr_derived_crash_4610.result.

This thing is present both before and after the patch:

explain select 1 from
(select f2, f3, val, count(id) from t4 join t2 left join t3 on 0) top
join t1 on f1 = f3 where f3 = 'aaaa' order by val;
id	select_type	table	type	possible_keys	key	key_len	ref	rows	Extra
1	PRIMARY	t1	const	PRIMARY	PRIMARY	12	const	1	
1	PRIMARY	<derived2>	ref	key0	key0	13	const	0	Using index condition; Using where; Using filesort

top-level select is an inner join, and there is a row with rows=0....
Would this cause "multiply-by-zero" wipe-out in further join plan choice?
(In this specific query it would not as there are only two tables in select #1... but what if there were more?)
...
Well, derived_cond_pushdown_innodb.result shows an example where we didn't have this before and now we do have it.

spetrunia · 2026-05-03T18:05:56Z

 --echo #

+
+#


Use --echo # so that the testcase is delimited in .result, too.

Ugh, fixed.

spetrunia · 2026-05-03T18:07:34Z

+  where ranked = 1
+)
+select * from ranked_table;
+


One has to strain the eye and compare $q0, $q1 ... . Please use short comment near let $q_N=... describing the difference.

Please do the same near EXPLAIN. So when/if that changes one can have a clue if that's a problem or not.

spetrunia · 2026-05-03T18:23:11Z

handle_single_part_rownumber:

    return (uint) records;
  }
  else
  {
    return (uint) 1;

Use ha_rows datatype for anything that's number of records.

mariadb-RexJohnston · 2026-05-03T23:38:14Z

I was just looking at the patch, mrr_derived_crash_4610.result.

This thing is present both before and after the patch:
explain select 1 from
(select f2, f3, val, count(id) from t4 join t2 left join t3 on 0) top
join t1 on f1 = f3 where f3 = 'aaaa' order by val;
id	select_type	table	type	possible_keys	key	key_len	ref	rows	Extra
1	PRIMARY	t1	const	PRIMARY	PRIMARY	12	const	1	
1	PRIMARY	<derived2>	ref	key0	key0	13	const	0	Using index condition; Using where; Using filesort
top-level select is an inner join, and there is a row with rows=0.... Would this cause "multiply-by-zero" wipe-out in further join plan choice? (In this specific query it would not as there are only two tables in select #1... but what if there were more?) ... Well, derived_cond_pushdown_innodb.result shows an example where we didn't have this before and now we do have it.

Unfortunately the number shown in the explain output is an integer, truncated from the double used in best_access_path. In the example in derived_cond_pushdown_innodb.result, rows is calculated from 0.2.
This is calculated from the number of records in the table (2, set in TABLE_LIST::fetch_number_of_rows because it's materialized) / MATCHING_ROWS_IN_OTHER_TABLE, set earlier in best_access_path as // Fix for small tables.
So, no. It won't cause a plan choice problem in this instance.

spetrunia · 2026-05-04T10:02:32Z

  longlong val_datetime_packed(THD *thd) override;
  uint size_of() const override { return sizeof *this; }
  Binlog_type_info binlog_type_info() const override;
+  bool hash_join_is_possible() override { return !null_bit; };      //MDEV-39369


We will need to discuss this change.
Topics:

Remove ; at the very end.

instead of null_bit, use real_maybe_null() (or should it be maybe_null() ?)

Why do we need to disable this? Because MDEV-39499 will make it possible to hit MDEV-39369 for a wider set of queries?

Should the change be in a separate patch?

I'm not sure what to do with this either. It's there because i needed to make a test failure involving a hash join go away. It should be a separate patch, but do we want this to be dependent on it?
The change allowing a const key part on a derived table will give the optimizer more choice, this is what has happened here.
I don't think this is a very good fix for MDEV-39369.

spetrunia · 2026-05-04T10:03:26Z

+      if ((*part_list->item)->type() != Item::NULL_ITEM)
+      {
+        if ((*part_list->item)->type() == Item::FIELD_ITEM)
+        {


What is the point of the above two nested if conditions?

if ((*part_list->item)->type() != Item::NULL_ITEM) if ((*part_list->item)->type() == Item::FIELD_ITEM) `` Can't one just check the second?

yes this is daft. I'll change it.

spetrunia · 2026-05-04T10:59:41Z

+      If we have only one select in our unit and
+        we have a window function in our select
+    */
+    if (!first->next_select() && first->with_sum_func)


To discuss: It is not clear that first->with_sum_func mean "we have a window function" ?
Is it the case here, because we've checked that derived table does not have SELECTS that have GROUP BY? (And if yes, where does the case of implicit grouping go?)

The Item_sum base class constructor sets this flag during parsing.

│ 11113 simple_window_func: │ 11114 ROW_NUMBER_SYM '(' ')' │ 11115 { │ > 11116 $$= new (thd->mem_root) Item_sum_row_number(thd); │ 11117 if (unlikely($$ == NULL)) │ 11118 MYSQL_YYABORT; │ 11119 }

This appears to be true of any window function, independent of any grouping operators.

spetrunia · 2026-05-04T12:17:08Z

If I just take this part of the patch:

diff --git a/sql/sql_select.cc b/sql/sql_select.cc
index 531b30cce21..5f5233aaf95 100644
--- a/sql/sql_select.cc
+++ b/sql/sql_select.cc
@@ -9002,8 +9002,18 @@ best_access_path(JOIN      *join,
                 table->opt_range[key].get_costs(&tmp);
                 goto got_cost2;
               }
-              /* quick_range couldn't use key! */
-              records= (double) s->records/rec;
+              /*
+               quick_range couldn't use key!
+               Check records per key, it may have been inferred from
+               derived table characteristics, so there will be no table map.
+              */
+
+              if (!(records=
+                    keyinfo->rec_per_key_null_aware(key_parts-1, notnull_part)))
+                records= (double) s->records/rec;
+
+              // /* quick_range couldn't use key! */
+              // records= (double) s->records/rec;
               if (unlikely(trace_access_idx.trace_started()))
                 trace_access_idx.
                   add("used_range_estimates", false).

Then I already get

Failing test(s): main.derived main.derived_cond_pushdown main.rowid_filter_myisam

Looking at the change in rowid_filter_myisam

--- /home/psergey/dev-git/11.4-review-derived-const-keys/mysql-test/main/rowid_filter_myisam.result     2026-05-
04 14:56:42.450322361 +0300
+++ /home/psergey/dev-git/11.4-review-derived-const-keys/mysql-test/main/rowid_filter_myisam.reject     2026-05-
04 15:02:59.218141505 +0300
@@ -91,7 +91,7 @@
 SELECT * FROM t1 HAVING (7, 9) IN (SELECT t2.i1, t2.i2 FROM t2 WHERE t2.i1 = 3);
 id     select_type     table   type    possible_keys   key     key_len ref     rows    filtered        Extra
 1      PRIMARY NULL    NULL    NULL    NULL    NULL    NULL    NULL    NULL    Impossible HAVING
-2      SUBQUERY        t2      index_subquery  i1,i2   i2      5       const   10      0.92    Using where
+2      SUBQUERY        NULL    NULL    NULL    NULL    NULL    NULL    NULL    NULL    no matching row in const
 table
 Warnings:
 Note   1003    /* select#1 */ select `test`.`t1`.`pk` AS `pk` from `test`.`t1` having 0
 DROP TABLE t1,t2;

Looks unexpected.

mariadb-RexJohnston · 2026-05-04T22:46:48Z

@@ -91,7 +91,7 @@
SELECT * FROM t1 HAVING (7, 9) IN (SELECT t2.i1, t2.i2 FROM t2 WHERE t2.i1 = 3);
id select_type table type possible_keys key key_len ref rows filtered Extra
1 PRIMARY NULL NULL NULL NULL NULL NULL NULL NULL Impossible HAVING
-2 SUBQUERY t2 index_subquery i1,i2 i2 5 const 10 0.92 Using where
+2 SUBQUERY NULL NULL NULL NULL NULL NULL NULL NULL no matching row in const
table
Warnings:
Note 1003 /* select#1 */ select test.t1.pk AS pk from test.t1 having 0
DROP TABLE t1,t2;
Looks unexpected.

t2 has 100 records per distinct value for both i1 and i2 for this test. The part of best_access_path() that ignores records_per_key for derived tables is what is updated here, and by not ignoring it, we change the number of records from an arbitrary 10 to an actual 100.
With only 10 records per key, the range rowid filter is discarded as being uneconomic. With 100 records, it isn't.
test_quick_select() returns an impossible range (t2.i1 = 3 and t2.i1 = 7), so causing a zero_result_cause in
JOIN::optimize_stage2(). This flows as a message into JOIN::save_explain_data_intern where the comment
/* Setting explain->message means that all other members are invalid */
explains the unexpected output.

..records per key Enabling derived keys optimization for derived.col = const pushed conditions. Estimating records per key in derived key for the optimizer based on form and/or size of components of a derived table. Consider any derived table of the form SELECT ..., ROW_NUMBER () OVER (PARTITION BY c1,c2 order by ...) FROM t1, t2, t3 ... WHERE ... If the optimizer generates a key on this derived table because of a constraint being pushed into it, it currently will not consider key components of the form col = const We lift this constraint and add code to TABLE::add_tmp_key to search for a window function ROW_NUMBER(). From the partition list c1, c2 we can in infer an estimate of the number of rows we expect to see for each key value. In the absence of EITS, we still have an number of expected records in our referred to table and will fall back to using that as a worst case scenario. As a consequence of forcing best_access_path to now consider a generated index's records per key on any type of derived table, we see a number of optimization changes, for example a range optimizer might now be considered best and might return IMPOSSIBLE_RANGE (seen in rowid_filter_myisam.result).

mariadb-RexJohnston force-pushed the bb-11.4-MDEV-39499 branch 3 times, most recently from 3b06e82 to 48a91d9 Compare April 29, 2026 21:01

gkodinov added the MariaDB Corporation label Apr 30, 2026

spetrunia self-requested a review May 3, 2026 14:37

spetrunia reviewed May 3, 2026

View reviewed changes

mariadb-RexJohnston force-pushed the bb-11.4-MDEV-39499 branch from 48a91d9 to fd9f165 Compare May 3, 2026 23:52

spetrunia reviewed May 4, 2026

View reviewed changes

mariadb-RexJohnston force-pushed the bb-11.4-MDEV-39499 branch from fd9f165 to f536bac Compare May 4, 2026 23:39

mariadb-RexJohnston force-pushed the bb-11.4-MDEV-39499 branch from f536bac to 6b3e95b Compare May 5, 2026 01:40

Uh oh!

Conversation

mariadb-RexJohnston commented Apr 29, 2026

Uh oh!

spetrunia commented May 3, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

spetrunia commented May 3, 2026

Uh oh!

mariadb-RexJohnston commented May 3, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mariadb-RexJohnston May 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

spetrunia commented May 4, 2026

Uh oh!

mariadb-RexJohnston commented May 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants

mariadb-RexJohnston May 4, 2026 •

edited

Loading

mariadb-RexJohnston commented May 4, 2026 •

edited

Loading