[feat](mtmv) support ALTER MATERIALIZED VIEW ADD COLUMN

[XnY-wei]

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:
Previously, users had to drop and recreate an asynchronous
materialized view (MTMV) whenever they needed an additional
derived column. This is expensive because it forces a full
rebuild and invalidates downstream query rewriting.

This PR introduces `ALTER MATERIALIZED VIEW ADD COLUMN

AS `, which lets users evolve the MTMV schema in place by appending a new column derived from an expression over the existing query. `CREATE TABLE orders ( o_orderkey bigint NOT NULL, o_custkey int NOT NULL, o_orderstatus varchar(1) NOT NULL, o_totalprice decimal(15,2) NOT NULL, o_orderdate date NOT NULL, o_orderpriority varchar(15) NOT NULL, o_clerk varchar(15) NOT NULL, o_shippriority int NOT NULL, o_comment varchar(79) NOT NULL ) ENGINE=OLAP DUPLICATE KEY(o_orderkey, o_custkey) PARTITION BY RANGE(o_orderdate) (PARTITION p_20240101 VALUES [('2024-01-01'), ('2024-02-01')), PARTITION p_20240201 VALUES [('2024-02-01'), ('2024-03-01'))) DISTRIBUTED BY HASH(o_orderkey) BUCKETS 16 PROPERTIES ( "replication_allocation" = "tag.location.default: 1", "min_load_replica_num" = "-1", "is_being_synced" = "false", "storage_medium" = "hdd", "storage_format" = "V2", "inverted_index_storage_format" = "V3", "light_schema_change" = "true", "disable_auto_compaction" = "false", "enable_single_replica_compaction" = "false", "group_commit_interval_ms" = "10000", "group_commit_data_bytes" = "134217728" );

mysql> CREATE MATERIALIZED VIEW orders_async_mv
-> BUILD IMMEDIATE
-> REFRESH AUTO
-> ON SCHEDULE EVERY 1 DAY
-> DISTRIBUTED BY RANDOM BUCKETS 2
-> PROPERTIES ('replication_num' = '1')
-> AS
-> SELECT
-> DATE_TRUNC(o_orderdate, 'MONTH') AS order_month,
-> SUM(o_totalprice) AS total_price
-> FROM orders
-> GROUP BY
-> DATE_TRUNC(o_orderdate, 'MONTH');
Query OK, 0 rows affected (0.11 sec)

mysql> INSERT INTO orders
-> (o_orderkey, o_custkey, o_orderstatus, o_totalprice, o_orderdate, o_orderpriority, o_clerk, o_shippriority, o_comment)
-> VALUES
-> (1, 100, 'O', 1000.00, '2024-01-01', '1-URGENT', 'Clerk#000000001', 1, 'first order'),
-> (2, 101, 'F', 2000.00, '2024-02-01', '2-HIGH', 'Clerk#000000002', 2, 'second order');
Query OK, 2 rows affected (0.21 sec)
{'label':'label_bbe60eaaa7dc43dc_831c715aeabb1684', 'status':'VISIBLE', 'txnId':'2013'}

mysql> select * from orders_async_mv;
Empty set (0.02 sec)

mysql> REFRESH MATERIALIZED VIEW orders_async_mv COMPLETE;
Query OK, 0 rows affected (0.02 sec)

mysql> select * from orders_async_mv;
+-------------+-------------+
| order_month | total_price |
+-------------+-------------+
| 2024-01-01 | 1000.00 |
| 2024-02-01 | 2000.00 |
+-------------+-------------+
2 rows in set (0.03 sec)
mysql> ALTER MATERIALIZED VIEW orders_async_mv ADD COLUMN total_orderkey INT NULL AS sum(o_orderke
y);
Query OK, 0 rows affected (0.01 sec)

mysql> select * from orders_async_mv;
+-------------+-------------+----------------+
| order_month | total_price | total_orderkey |
+-------------+-------------+----------------+
| 2024-01-01 | 1000.00 | NULL |
| 2024-02-01 | 2000.00 | NULL |
+-------------+-------------+----------------+
2 rows in set (0.05 sec)

mysql> REFRESH MATERIALIZED VIEW orders_async_mv COMPLETE;
Query OK, 0 rows affected (0.00 sec)

mysql> select * from orders_async_mv;
+-------------+-------------+----------------+
| order_month | total_price | total_orderkey |
+-------------+-------------+----------------+
| 2024-01-01 | 1000.00 | 1 |
| 2024-02-01 | 2000.00 | 2 |
+-------------+-------------+----------------+
2 rows in set (0.03 sec)`

Release note

None

Check List (For Author)

• Test

  • Regression test
  • Unit Test
  • Manual test (add detailed scripts or steps below)
  • No need to test or manual test. Explain why:
    • This is a refactor/code format and no logic has been changed.
    • Previous test can cover this change.
    • No code files have been changed.
    • Other reason

• Behavior changed:

  • No.
  • Yes.

• Does this need documentation?

  • No.
  • Yes.

Check List (For Reviewer who merge this PR)

• Confirm the release note
• Confirm test cases
• Confirm document
• Add branch pick label

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[morrySnow]

@yujun777 please review this PR

[morrySnow]

Thanks for this PR! Adding ALTER MATERIALIZED VIEW ADD COLUMN is a valuable feature. I found several issues that should be addressed, most critically a compilation error and a missing cache invalidation that would cause stale query rewrite results.

Summary of findings (most severe first):

1. Compilation error: Env.notifyTableMetaChange() does not exist anywhere in the codebase.
2. Missing cache invalidation: alterAddColumn() does not increment rewriteCacheGeneration or clear cacheWithGuard/cacheWithoutGuard.
3. Accidental library downgrade: MapUtils import changed from collections4 to collections (v3).
4. UNION ALL silently broken: Only the first branch gets the new column, causing column count mismatch.
5. Code duplication: SelectColumnClauseIndexFinder duplicates BaseViewInfo.IndexFinder.
6. NPE risk: rewriteQuerySqlForAddColumn NPEs when AST has no SelectColumnClause.
7. Exception escape: alterAddColumn() RuntimeException bypasses processAlterMTMV's catch(UserException) handler.
8. SQL rewriting duplicates existing utility: Pattern already exists in BaseViewInfo.rewriteProjectsToUserDefineAlias.

See individual inline comments for details.

[morrySnow]

CRITICAL — Compilation error: Env.getCurrentEnv().notifyTableMetaChange(mtmv) calls a method that does not exist anywhere in the codebase. A full-project grep for notifyTableMetaChange returns zero results.

This code will not compile. The intended behavior (per the comment "keep FE-side plan/cache consistent after MTMV schema change") should be achieved by:

1. Removing this call entirely, AND
2. Adding this.rewriteCacheGeneration++ and clearing cacheWithGuard/cacheWithoutGuard inside MTMV.alterAddColumn() itself — following the pattern already used at MTMV.java:496-498 in processBaseViewChange().

Alternatively, if a new Env method is truly needed, it must be added to Env.java with the appropriate implementation.

[morrySnow]

Thanks for this PR! Adding ALTER MATERIALIZED VIEW ADD COLUMN is a valuable feature. I found several issues that should be addressed, most critically a compilation error and a missing cache invalidation that would cause stale query rewrite results.

Summary of findings (most severe first):

1. Compilation error: Env.notifyTableMetaChange() does not exist anywhere in the codebase.
2. Missing cache invalidation: alterAddColumn() does not increment rewriteCacheGeneration or clear cacheWithGuard/cacheWithoutGuard.
3. Accidental library downgrade: MapUtils import changed from collections4 to collections (v3).
4. UNION ALL silently broken: Only the first branch gets the new column, causing column count mismatch.
5. Code duplication: SelectColumnClauseIndexFinder duplicates BaseViewInfo.IndexFinder.
6. NPE risk: rewriteQuerySqlForAddColumn NPEs when AST has no SelectColumnClause.
7. Exception escape: alterAddColumn() RuntimeException bypasses processAlterMTMV's catch(UserException) handler.
8. SQL rewriting duplicates existing utility: Pattern already exists in BaseViewInfo.rewriteProjectsToUserDefineAlias.

See individual inline comments for details.

[morrySnow]

BUG — Missing rewrite cache invalidation: alterAddColumn() increments schemaChangeVersion and resets refreshSnapshot, but does not increment rewriteCacheGeneration or clear cacheWithGuard / cacheWithoutGuard.

Between the ALTER ADD COLUMN and the next successful refresh, the MTMV's rewrite caches will serve stale results to query rewriting — matching incoming queries against the old schema and potentially returning incorrect results.

The existing pattern in processBaseViewChange() (lines 496-498) already shows the correct approach:

rewriteCacheGeneration++;
cacheWithGuard = null;
cacheWithoutGuard = null;

Add these three lines here alongside schemaChangeVersion++.

[morrySnow]

Thanks for this PR! Adding ALTER MATERIALIZED VIEW ADD COLUMN is a valuable feature. I found several issues that should be addressed, most critically a compilation error and a missing cache invalidation that would cause stale query rewrite results.

Summary of findings (most severe first):

1. Compilation error: Env.notifyTableMetaChange() does not exist anywhere in the codebase.
2. Missing cache invalidation: alterAddColumn() does not increment rewriteCacheGeneration or clear cacheWithGuard/cacheWithoutGuard.
3. Accidental library downgrade: MapUtils import changed from collections4 to collections (v3).
4. UNION ALL silently broken: Only the first branch gets the new column, causing column count mismatch.
5. Code duplication: SelectColumnClauseIndexFinder duplicates BaseViewInfo.IndexFinder.
6. NPE risk: rewriteQuerySqlForAddColumn NPEs when AST has no SelectColumnClause.
7. Exception escape: alterAddColumn() RuntimeException bypasses processAlterMTMV's catch(UserException) handler.
8. SQL rewriting duplicates existing utility: Pattern already exists in BaseViewInfo.rewriteProjectsToUserDefineAlias.

See individual inline comments for details.

[morrySnow]

Code duplication: SelectColumnClauseIndexFinder (lines 144-178) is a near-identical copy of BaseViewInfo.IndexFinder (BaseViewInfo.java:215-259). Both extend DorisParserBaseVisitor<Void>, both override visitChildren, visitCte, and visitSelectColumnClause with the same pattern.

The only difference is that BaseViewInfo.IndexFinder also captures namedExpressionContexts — but SelectColumnClauseIndexFinder could simply extend or wrap the existing IndexFinder instead of duplicating it.

Extract a shared SelectClauseFinder utility or have SelectColumnClauseIndexFinder delegate to the existing BaseViewInfo.IndexFinder.

[morrySnow]

Thanks for this PR! Adding ALTER MATERIALIZED VIEW ADD COLUMN is a valuable feature. I found several issues that should be addressed, most critically a compilation error and a missing cache invalidation that would cause stale query rewrite results.

Summary of findings (most severe first):

1. Compilation error: Env.notifyTableMetaChange() does not exist anywhere in the codebase.
2. Missing cache invalidation: alterAddColumn() does not increment rewriteCacheGeneration or clear cacheWithGuard/cacheWithoutGuard.
3. Accidental library downgrade: MapUtils import changed from collections4 to collections (v3).
4. UNION ALL silently broken: Only the first branch gets the new column, causing column count mismatch.
5. Code duplication: SelectColumnClauseIndexFinder duplicates BaseViewInfo.IndexFinder.
6. NPE risk: rewriteQuerySqlForAddColumn NPEs when AST has no SelectColumnClause.
7. Exception escape: alterAddColumn() RuntimeException bypasses processAlterMTMV's catch(UserException) handler.
8. SQL rewriting duplicates existing utility: Pattern already exists in BaseViewInfo.rewriteProjectsToUserDefineAlias.

See individual inline comments for details.

[morrySnow]

NPE risk: SelectColumnClauseIndexFinder.getIndex() returns null if the AST contains no SelectColumnClause (e.g., VALUES statement, or a parse tree where visitSelectColumnClause is never invoked). At line 135, selectColumnClauseIndex.first will throw NullPointerException.

This is caught by alterAddColumn()'s catch (Exception) and wrapped as RuntimeException, but the error message ("alter materialized view add column failed") gives no indication that a null index was the cause.

Add a null check before line 135:

if (selectColumnClauseIndex == null) {
    throw new IllegalStateException("Could not locate SELECT column clause in query SQL");
}

[morrySnow]

Thanks for this PR! Adding ALTER MATERIALIZED VIEW ADD COLUMN is a valuable feature. I found several issues that should be addressed, most critically a compilation error and a missing cache invalidation that would cause stale query rewrite results.

Summary of findings (most severe first):

1. Compilation error: Env.notifyTableMetaChange() does not exist anywhere in the codebase.
2. Missing cache invalidation: alterAddColumn() does not increment rewriteCacheGeneration or clear cacheWithGuard/cacheWithoutGuard.
3. Accidental library downgrade: MapUtils import changed from collections4 to collections (v3).
4. UNION ALL silently broken: Only the first branch gets the new column, causing column count mismatch.
5. Code duplication: SelectColumnClauseIndexFinder duplicates BaseViewInfo.IndexFinder.
6. NPE risk: rewriteQuerySqlForAddColumn NPEs when AST has no SelectColumnClause.
7. Exception escape: alterAddColumn() RuntimeException bypasses processAlterMTMV's catch(UserException) handler.
8. SQL rewriting duplicates existing utility: Pattern already exists in BaseViewInfo.rewriteProjectsToUserDefineAlias.

See individual inline comments for details.

[morrySnow]

Exception escape: MTMV.alterAddColumn() wraps all exceptions in RuntimeException (see catch (Exception e) at the end of the method). However, processAlterMTMV only catches UserException at line 1336, not RuntimeException.

If alterAddColumn() fails (e.g., concurrent modification, corrupted index metadata, or the NPE from Finding 6 above), the RuntimeException will propagate uncaught past the processAlterMTMV handler, potentially crashing the statement execution.

Either:

1. Make alterAddColumn() throw a checked exception that processAlterMTMV can handle, or
2. Add a catch (RuntimeException e) in processAlterMTMV for this case, or
3. Catch RuntimeException as well as UserException at line 1336.

[morrySnow]

Thanks for this PR! Adding ALTER MATERIALIZED VIEW ADD COLUMN is a valuable feature. I found several issues that should be addressed, most critically a compilation error and a missing cache invalidation that would cause stale query rewrite results.

Summary of findings (most severe first):

1. Compilation error: Env.notifyTableMetaChange() does not exist anywhere in the codebase.
2. Missing cache invalidation: alterAddColumn() does not increment rewriteCacheGeneration or clear cacheWithGuard/cacheWithoutGuard.
3. Accidental library downgrade: MapUtils import changed from collections4 to collections (v3).
4. UNION ALL silently broken: Only the first branch gets the new column, causing column count mismatch.
5. Code duplication: SelectColumnClauseIndexFinder duplicates BaseViewInfo.IndexFinder.
6. NPE risk: rewriteQuerySqlForAddColumn NPEs when AST has no SelectColumnClause.
7. Exception escape: alterAddColumn() RuntimeException bypasses processAlterMTMV's catch(UserException) handler.
8. SQL rewriting duplicates existing utility: Pattern already exists in BaseViewInfo.rewriteProjectsToUserDefineAlias.

See individual inline comments for details.

[morrySnow]

Duplicated SQL-rewriting pattern: rewriteQuerySqlForAddColumn() (lines 131-141) follows the same AST-then-StringUtils.overlay pattern already used in BaseViewInfo.rewriteProjectsToUserDefineAlias() (BaseViewInfo.java:142-167). Additionally, BaseViewInfo.rewriteSql() (BaseViewInfo.java:129) provides a general-purpose position-keyed SQL rewriting primitive (TreeMap<Pair<Integer,Integer>, String> → rewritten SQL).

There are now three copies of this pattern in the codebase (MTMV, BaseViewInfo, and LogicalPlanBuilderForSyncMv). Consider extracting a shared SQLRewriteUtils utility to avoid triple maintenance of the same logic.

[morrySnow]

test

[morrySnow]

test

[morrySnow]

test

[morrySnow]

test

[morrySnow]

test

[morrySnow]

test

[morrySnow]

test

[morrySnow]

test

[morrySnow]

test

[morrySnow]

test

[morrySnow]

test

[morrySnow]

test

[morrySnow]

test

[morrySnow]

test

[morrySnow]

Accidental library downgrade: The import changed from org.apache.commons.collections4.MapUtils (commons-collections4 v4.x, the version explicitly declared in fe/pom.xml) to org.apache.commons.collections.MapUtils (commons-collections v3.x, not declared as a direct dependency).

• v4 provides generic type-safe MapUtils.isEmpty(Map<K,V>); v3 takes raw Map.
• The entire rest of the codebase uses org.apache.commons.collections4.
• This appears to be an accidental IDE auto-import — the MapUtils.isEmpty() calls at lines 781/787 are unchanged and unrelated to the ADD COLUMN feature.

Fix: Revert this import to org.apache.commons.collections4.MapUtils.

[morrySnow]

UNION ALL silently produces broken SQL: The SelectColumnClauseIndexFinder only finds the first SELECT clause. For UNION ALL queries like SELECT a FROM t1 UNION ALL SELECT a FROM t2, only the first branch gets the new column appended, producing a column-count mismatch error at refresh time.

The test documents this as a known limitation (⚠️) but ships the code without any guard. Consider either:

1. Rejecting the ALTER at analyze time if the query contains UNION/INTERSECT/EXCEPT (with a clear error message to the user), OR
2. Walking ALL selectColumnClause nodes (not just the first), applying the overlay to each branch — this is how StarRocks handles it with AST2SQLVisitor.